DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents

# DocsChisel: Adaptive Tool Documentation Optimization Framework for LLM Agents
Source: [https://arxiv.org/html/2608.10037](https://arxiv.org/html/2608.10037)
###### Abstract

Large language models \(LLMs\) increasingly rely on external tools to accomplish complex real\-world tasks, making tool documentation a critical grounding resource for LLM agents\. Existing studies mainly focus on improving thetool\-usecapabilities of LLM agents, while largely treating tool documentation as a fixed input\. Although several recent works attempt to optimize tool documentation through rewriting or compression, little is known about how the information contained in tool documentation affects agent performance across different settings\.

To bridge this gap, we conduct a large\-scale empirical study on tool documentation for LLM agents\. Our study reveals substantial heterogeneity in the informationfieldsprovided by existing tool documentation\. Moreover, the effectiveness of different information fields is highly dependent on the task domain, LLM backbone, and agent paradigm, indicating that no fixed tool documentation can consistently generalize across diverse agent settings\.

Motivated by these findings, we proposeDocsChisel, an adaptive tool documentation optimization framework for LLM agents\.DocsChiselanalyzes failed execution traces of a target LLM agent to identify documentation\-related issues, and iteratively optimizes tool documentation by adding, removing, and refining information fields for each tool\. We evaluateDocsChiselagainst two state\-of\-the\-art baselines,i\.e\.,EasyToolandDRAFT\. Experimental results show thatDocsChiselimproves the task successrateof LLM agents by95\.89%over the original tool documentation and by75\.15%, on average, over existing baselines, while incurring limited optimization time and token overhead\.

## IIntroduction

Large language models \(LLMs\)\[[34](https://arxiv.org/html/2608.10037#bib.bib2),[12](https://arxiv.org/html/2608.10037#bib.bib3)\]have demonstrated strong capabilities across diverse tasks\[[18](https://arxiv.org/html/2608.10037#bib.bib4),[11](https://arxiv.org/html/2608.10037#bib.bib5),[20](https://arxiv.org/html/2608.10037#bib.bib6)\], such as code generation, question answering, and data analysis\. To execute real\-world tasks, LLMs increasingly rely on external tools,e\.g\.,web browsers, code interpreters, and file systems, leading to the emergence oftool\-useLLM agents \(referred to as LLM agents hereafter\)\[[56](https://arxiv.org/html/2608.10037#bib.bib21),[13](https://arxiv.org/html/2608.10037#bib.bib7),[54](https://arxiv.org/html/2608.10037#bib.bib8),[44](https://arxiv.org/html/2608.10037#bib.bib9),[38](https://arxiv.org/html/2608.10037#bib.bib10),[40](https://arxiv.org/html/2608.10037#bib.bib11)\]\. Emerging LLM agent ecosystems, including agent development platforms\[[4](https://arxiv.org/html/2608.10037#bib.bib12),[7](https://arxiv.org/html/2608.10037#bib.bib13)\], tool hosting platforms\[[43](https://arxiv.org/html/2608.10037#bib.bib14),[37](https://arxiv.org/html/2608.10037#bib.bib15)\], and agent evaluation suites\[[53](https://arxiv.org/html/2608.10037#bib.bib50),[27](https://arxiv.org/html/2608.10037#bib.bib46)\], provide large collections of reusable tools and standardized tool interfaces\. These tools are typically accompanied by tool documentation that includes several information fields about their functionalities, invocation constraints, and other details, helping LLM agents understand and invoke tools correctly\.

Traditional API documentation has long been studied in software engineering, primarily to improve its readability, completeness, and usability for developers\[[6](https://arxiv.org/html/2608.10037#bib.bib34),[5](https://arxiv.org/html/2608.10037#bib.bib40),[30](https://arxiv.org/html/2608.10037#bib.bib37),[33](https://arxiv.org/html/2608.10037#bib.bib41),[46](https://arxiv.org/html/2608.10037#bib.bib38),[24](https://arxiv.org/html/2608.10037#bib.bib42),[39](https://arxiv.org/html/2608.10037#bib.bib35),[32](https://arxiv.org/html/2608.10037#bib.bib36),[49](https://arxiv.org/html/2608.10037#bib.bib39)\]\. However, tool documentation for LLM agents differs in both audience and content\. The documentation consumers are LLM agents that must infer tool functionality and generate executable toolinvocationsdirectly from in\-context documentation\. Meanwhile, the documentation emphasis shifts from general API understanding to task\-oriented tool use, requiring information such as usage conditions, parameter semantics, invocation constraints, and output interpretation\. As such documentation guides runtime tool selection and invocation, ambiguity or missing information can lead to incorrect toolinvocationsand cascading task failures\. Thus, tool documentation should be viewed not only as a human\-readable reference, but also as a critical grounding resource whose quality heavily affects the task success rate of LLM agents\[[19](https://arxiv.org/html/2608.10037#bib.bib16),[58](https://arxiv.org/html/2608.10037#bib.bib56),[40](https://arxiv.org/html/2608.10037#bib.bib11)\]\.

Literature\.Despite the importance of tool documentation, existing studies\[[25](https://arxiv.org/html/2608.10037#bib.bib20),[41](https://arxiv.org/html/2608.10037#bib.bib17),[60](https://arxiv.org/html/2608.10037#bib.bib18),[22](https://arxiv.org/html/2608.10037#bib.bib51),[55](https://arxiv.org/html/2608.10037#bib.bib44),[8](https://arxiv.org/html/2608.10037#bib.bib45),[27](https://arxiv.org/html/2608.10037#bib.bib46),[45](https://arxiv.org/html/2608.10037#bib.bib48),[47](https://arxiv.org/html/2608.10037#bib.bib19)\]mainly focus on adding or improvingtool\-usecapabilities of LLM agents from the perspectives of tool construction, tool retrieval, tool invocation, and evaluation\. While these studies have significantly advanced the development of LLM agents, they usually treat tool documentation as a fixed given input, without systematically examining how documentation itself affects LLM agent behavior and task successrate\. Only a few recent studies\[[10](https://arxiv.org/html/2608.10037#bib.bib60),[9](https://arxiv.org/html/2608.10037#bib.bib61),[29](https://arxiv.org/html/2608.10037#bib.bib62),[58](https://arxiv.org/html/2608.10037#bib.bib56),[14](https://arxiv.org/html/2608.10037#bib.bib58),[42](https://arxiv.org/html/2608.10037#bib.bib57)\]explicitly optimize tool documentation for LLM agents\. For instance,EasyTool\[[58](https://arxiv.org/html/2608.10037#bib.bib56)\]standardizes diverse tool documentation into concise tool instructions with a unified template, thereby reducing the context overhead caused by documentation, whereasDRAFT\[[42](https://arxiv.org/html/2608.10037#bib.bib57)\]iterativelyrefines tool documentation based onoutputs of external tools\. However, these approaches mostly optimize tool documentation within existing information fields through rewriting, correction, standardization, or compression\. They pay less attention to whether the documentation fields themselves are sufficient, redundant, or suitable for different LLM agents\.

In practice, tool documentation across LLM agent ecosystems may differ substantially in the information fields it provides\.However, it remains unclear whether the utility of these information fields varies across agent settings\. For example, do agents powered by less capable LLM backbones benefit more from explicit usage examples? Do multi\-agent paradigms place greater demands on usage guidance and invocation constraints?These observations raise a fundamental question, i\.e\.,what information fields are provided by existing tool documentation, and how do these information fields affect the task success rate of LLM agents under different agent settings?

TABLE I:Overview of the Collected Tool\-Use DatasetsDataset\#ToolsDescriptionWorkBench\[[48](https://arxiv.org/html/2608.10037#bib.bib52)\]26A tool\-use LLM agent benchmark with sandboxes, realistic office tasks, databases, and executable tools\.API\-Bank\[[27](https://arxiv.org/html/2608.10037#bib.bib46)\]2,211A comprehensive evaluation benchmark with diverse APIs and multi\-turn dialogues for LLM agents\.ToolLLM\[[40](https://arxiv.org/html/2608.10037#bib.bib11)\]16,464A large\-scale suite of real\-world APIs for evaluating multi\-step tool invocations by LLM agents\.APIBench\[[38](https://arxiv.org/html/2608.10037#bib.bib10)\]1,645A massive API corpus for studying LLM connectivity to thousands of remote APIs\.ToolAlpaca\[[50](https://arxiv.org/html/2608.10037#bib.bib49)\]426A simulated dataset with diverse tool APIs and structured tool documentation for training LLMs\.AnyToolBench\[[8](https://arxiv.org/html/2608.10037#bib.bib45)\]13A benchmark for hierarchical, large\-scale tool calling with self\-reflective LLM agents\.ToolBench\[[53](https://arxiv.org/html/2608.10037#bib.bib50)\]232A benchmark for evaluating open\-source LLMs on tool manipulation and invocation tasks\.ShortcutsBench\[[45](https://arxiv.org/html/2608.10037#bib.bib48)\]1,414A large\-scale benchmark based on Apple Shortcuts, containing real APIs, user queries, and action sequences\.WildToolBench\[[57](https://arxiv.org/html/2608.10037#bib.bib53)\]1,600A tool\-use benchmark grounded in real\-world user behavior patterns and realistic multi\-step scenarios\.SWE\-bench\[[23](https://arxiv.org/html/2608.10037#bib.bib43),[54](https://arxiv.org/html/2608.10037#bib.bib8)\]15A software engineering benchmark requiring LLM agents to resolve real GitHub issues via repository tools\.CRMArena\[[21](https://arxiv.org/html/2608.10037#bib.bib47)\]27A domain\-specific benchmark for professional CRM tasks in realistic enterprise environments\.τ\\tau\-bench\[[55](https://arxiv.org/html/2608.10037#bib.bib44)\]28A benchmark for interactions among tools, agents and users with domain\-specific tools and dialogues\.TooLe\[[22](https://arxiv.org/html/2608.10037#bib.bib51)\]390A meta\-tool benchmark for deciding whether to invoke tools and selecting appropriate tools\.ToolLens\[[41](https://arxiv.org/html/2608.10037#bib.bib17)\]464A tool\-retrieval dataset designed for multi\-tool scenarios, emphasizing complete and diverse tool selection\.

Empirical Study\.To understand tool documentation for LLM agents and its impact on task success rate, we conduct a large\-scale empirical study across 14 tool\-use datasets collected from LLM agent ecosystems\. Specifically, we investigate their information field composition, and evaluate the task success rate of two different LLM agent paradigms \(i\.e\.,ReAct\[[56](https://arxiv.org/html/2608.10037#bib.bib21)\]and Multi\-Agent\[[51](https://arxiv.org/html/2608.10037#bib.bib22)\]\) on WorkBench\[[48](https://arxiv.org/html/2608.10037#bib.bib52)\], using three different LLM backbones \(i\.e\.,GPT\-4o\[[35](https://arxiv.org/html/2608.10037#bib.bib23)\], GLM\-5\[[59](https://arxiv.org/html/2608.10037#bib.bib24)\]and Claude Haiku 4\.5\[[3](https://arxiv.org/html/2608.10037#bib.bib25)\]\)\. We design the following research questions\.

- •RQ1 Composition Analysis\.What information fields are commonly contained in different tooldocumentation?
- •RQ2 Impact Analysis\.How do different information fields affect the task success rate of LLM agents across different task domains, LLM backbones, and agent paradigms?

Our results reveal that existing tool documentation in agent ecosystems exhibits substantial heterogeneity in field composition, while different information fields have different impacts on task success rate of LLM agents across task domains, LLM backbones, and agent paradigms\.Specifically, we identify 17 information fields, among which onlytool nameandfunctionality descriptionare provided by all 14 datasets, whileusage guidanceandinvocation constraintappear in only two datasets\. Moreover, the same information field may have different or even opposite effects under different agent settings\. On average, adding or removing a single information field changes the task success rate of LLM agents by 6\.34 percentage points in absolute terms\.These findings demonstrate that fixed tool documentation cannot generalize well across different LLM agent settings and should be optimized adaptively\.

Our Approach\.Motivated by these insights, we proposeDocsChisel, an adaptive tool documentation optimization framework for LLM agents\. Specifically, given a target LLM agent,DocsChiselfirst executes representative tool\-use queries with the original tool documentation, and collects failed agent execution traces\. Then,DocsChiselanalyzes these failed traces to identify documentation\-related issues, and uses LLMs to iteratively revise tool documentation\. Instead of simply rewriting tool documentation into a shorter or more general form,DocsChiseloptimizes both the content and structure of tool documentation by adding, removing, or refining information fields for each tool, thereby generating adaptive tool documentation for different LLM agent settings to improve both tool invocation correctness and task success rate\.

Evaluation\.We implement a prototype ofDocsChisel, and conduct experimentson 74 tools across 9 task domains selected from WorkBench\[[48](https://arxiv.org/html/2608.10037#bib.bib52)\]and API\-Bank\[[27](https://arxiv.org/html/2608.10037#bib.bib46)\]to demonstrate its effectiveness and efficiency, compared with two state\-of\-the\-art baselines,i\.e\.,EasyTool\[[58](https://arxiv.org/html/2608.10037#bib.bib56)\]and DRAFT\[[42](https://arxiv.org/html/2608.10037#bib.bib57)\]\. Experimental results demonstrate thatDocsChiselconsistently improves task success rate under diverse agent settings by95\.89%compared with using the original tool documentation, and achieves an average of75\.15%improvement compared with baselines\.Besides,DocsChiselmaintains practical optimization efficiency, requiring 12\.65 minutes per tool, on average, to optimize the tool documentation for LLM agents\.

Contribution\.This work makes the following contributions\.

- •We conduct a systematic empirical study on tool documentation for LLM agents, revealing the varying effectiveness of different information fields across different agent settings\.
- •We design and implement an adaptive tool documentation optimization framework,DocsChisel, for LLM agents, thereby improving the tool invocation correctness and task success rate of LLM agents\.
- •We conduct extensive experiments to demonstrate the effectiveness and efficiency ofDocsChisel\.

## IIEmpirical Study

To systematically investigate the two research questions introduced in Sec\.[I](https://arxiv.org/html/2608.10037#S1), we present a large\-scale empirical study on the information field composition of different tool documentation, and evaluate the task success rate of different LLM agent settings with different tool documentation\.

### II\-AStudy Design

Documentation Collection\.We collect tool documentation from 14 representative datasets shown in Table[I](https://arxiv.org/html/2608.10037#S1.T1), covering diverse tool\-use scenarios,e\.g\.,workflow execution, mobile automation, and software engineering tasks\. We select these datasets based on the following criteria:\(1\) providing rich tool collections and corresponding tool documentation, \(2\) covering different task domains and execution environments, \(3\) having strong community influence and being widely used in recent studies on LLM agents\.In total, we collect24,955tools along with their corresponding tool documentation\.

LLM Selection and Agent Paradigm Setting\.To evaluate the effectiveness of tool documentation under different LLM agent settings, we select three representative LLMs as agent backbones,i\.e\.,GPT\-4o\[[35](https://arxiv.org/html/2608.10037#bib.bib23)\], GLM\-5\[[59](https://arxiv.org/html/2608.10037#bib.bib24)\], and Claude Haiku 4\.5\[[3](https://arxiv.org/html/2608.10037#bib.bib25)\], covering diverse model families widely used in recent LLM agents\. We further instantiate two agent paradigms,i\.e\.,a ReAct agent implemented with LangChain\[[25](https://arxiv.org/html/2608.10037#bib.bib20)\], and a Multi\-Agent system implemented with AutoGen\[[51](https://arxiv.org/html/2608.10037#bib.bib22)\]\.

Evaluation Metric\.We usetask success rate\(TS\) to measure whether an LLM agent successfully completes user queries\. Given a test suite containing tool\-use queries𝒬\\mathcal\{Q\}, for each queryqi∈𝒬\{q\_\{i\}\}\\in\\mathcal\{Q\}, we denote its execution outcome ass​\(qi\)s\(q\_\{i\}\)\.s​\(qi\)=1s\(q\_\{i\}\)=1indicates that the task is correctly completed, ands​\(qi\)=0s\(q\_\{i\}\)=0otherwise\. We compute the task success rate asTS=1\|𝒬\|​∑i=1\|𝒬\|𝕀​\(s​\(qi\)=1\)\\text\{TS\}=\\frac\{1\}\{\|\\mathcal\{Q\}\|\}\\sum\_\{i=1\}^\{\|\\mathcal\{Q\}\|\}\\mathbb\{I\}\(s\(q\_\{i\}\)=1\), where𝕀​\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function\.

RQ Setup\.ForRQ1, we manually inspect the tool documentation from the 14 datasets we collected in Table[I](https://arxiv.org/html/2608.10037#S1.T1)\. Because these datasets use different formats and inconsistent information field names, three authors independently extracted candidate information fields, and then aligned semantically equivalent information through discussion using unified names following the most commonly used field names in the collected datasets\. Disagreements were resolved by consensus to reduce individual bias\. Based on the final aligned taxonomy, we report the coverage ratio of each type of information field across datasets\.

ForRQ2, we conduct the empirical evaluation using the tool documentation provided in WorkBench\[[48](https://arxiv.org/html/2608.10037#bib.bib52)\], which contains 26 tools from five task domains,i\.e\.,data analysis, email management, calendar management, project management, and customer relationship management \(CRM\)\.We modify the tool documentation by removing or adding one information field at a time while keeping all remaining fields unchanged according to the taxonomy derived fromRQ1\. Specifically, if the target information field already exists in the tool documentation, we remove it to measure the impact of missing information\. Otherwise, we manually supplement the information field based on the original documentation and dataset\-provided metadata,e\.g\.,tool code and correct execution outcome, to measure the impact of additional information\.Based on these documentation variants, we conduct controlled experiments to investigate whether the effectiveness of information fields remains consistent across different task domains, LLM backbones, and agent paradigms\.

First, to study whether the impact varies across task domains, we use GPT\-4o with the ReAct agent, and evaluate documentation variants on the five task domains in WorkBench\. Second, to study whether the impact varies across LLM backbones, we focus on the data analysis domain and the ReAct agent, and repeat the experiments with three different LLM backbones,i\.e\.,GPT\-4o, GLM\-5, and Claude Haiku 4\.5\. Third, to study whether the impact varies across agent paradigms, we fix the data analysis domain and GPT\-4o, and compare the results under the ReAct and Multi\-Agent paradigms\. In all experiments, we keep the query set, prompt template, and execution environment unchanged within each comparison\.

TABLE II:Prevalence of Information Fields across DatasetsInformation FieldBrief DescriptionPrevalenceTool Name \(TN\)Invocation identifier of target tool14/14Functionality Description \(FD\)Tool purpose and capability14/14Task Domain \(TD\)Application scenario or category6/14Input Parameter Name \(IPN\)Input argument identifier13/14Input Parameter Description \(IPD\)Input argument semantics12/14Input Parameter Type \(IPT\)Input data types or formats13/14Default Parameter Value \(DPV\)Default input parameter values3/14Parameter Required Flag \(PRF\)Required argument indicator10/14Parameter Optional Flag \(POF\)Optional argument indicator7/14Return Parameter Name \(RPN\)Output argument identifier4/14Return Parameter Description \(RPD\)Output argument semantics5/14Return Parameter Type \(RPT\)Output data types or formats5/14Code Implementation \(CI\)Tool implementation snippets5/14Usage Example \(UE\)Concrete invocation examples4/14Response Template \(RT\)Expected tool response format4/14Usage Guidance \(UG\)Tool usage recommendations2/14Invocation Constraint \(IC\)Invocation conditions or restrictions2/14

### II\-BComposition Analysis \(RQ1\)

After manually analyzing the collected tool documentation, we observe that tools within the same dataset usually follow a consistent documentation convention,i\.e\.,tools in the same dataset tend to contain the same set of information fields\. However, such conventions vary across datasets, leading to different field compositions in existing tool documentation\.

We identify 17 common information fields, and report their prevalence across the 14 datasets\. As shown in Table[II](https://arxiv.org/html/2608.10037#S2.T2),fields describing tool identity and input schemas are widely adopted\.Tool nameandfunctionality descriptionappear in all datasets, whileinput parameter name,input parameter description, andinput parameter typeappear in most datasets\. In contrast, fields supporting task\-oriented tool use and result interpretation, such asdefault parameter value,return parameter name,return parameter type,usage exampleandresponse template, are provided much less consistently\.Usage guidanceandinvocation constraintappear in only two datasets\.

Findings\.Existing tool documentation shows cross\-dataset heterogeneity in information field composition\. Although tools within the same dataset follow a consistent convention, different datasets provide different sets of information fields\.

### II\-CImpact Analysis \(RQ2\)

Fig\.[1](https://arxiv.org/html/2608.10037#S2.F1)reports the change inTSafter adding or removing one information field from the original tool documentation\. Black x\-axis labels denote removed fields, while red labels denote added fields\. As shown in Fig\.[1a](https://arxiv.org/html/2608.10037#S2.F1.sf1)\-[1c](https://arxiv.org/html/2608.10037#S2.F1.sf3),under GPT\-4o with ReAct, the evaluated fields exhibit both consistent and domain\-dependent effects\. Among the 17 fields, only 6 fields show the same effect direction across all three domains\. Removingtool name\(TN\),task domain\(TD\),return parameter type\(RPT\),response template\(RT\), orparameter optional flag\(POF\) consistently decreasesTS, whereas addinginvocation constraint\(IC\) consistently improves it\. For example, removing TD decreasesTSby 3\.75 to 13\.75 percentage points, while adding IC improves it by 0\.75 to 11\.25 percentage points\. In contrast, the remaining fields show domain\-dependent effects\. Addingparameter required flag\(PRF\), for instance, changesTSby \+2\.50, \-5\.61, and \+0\.75 percentage points in data analysis, email management, and calendar management, respectively\. Due to space limitations, results for project management and customer relationship management are provided on our website\[[1](https://arxiv.org/html/2608.10037#bib.bib1)\]\.

The effects also vary across LLM backbones\. As shown in Fig\.[1a](https://arxiv.org/html/2608.10037#S2.F1.sf1), Fig\.[1d](https://arxiv.org/html/2608.10037#S2.F1.sf4), and Fig\.[1e](https://arxiv.org/html/2608.10037#S2.F1.sf5), when fixing the domain and agent paradigm, 12 of the 17 fields exhibit different effect directions across GPT\-4o, GLM\-5, and Claude Haiku 4\.5\. For example, removing POF decreasesTSfor GPT\-4o and GLM\-5, but improves it by 7\.5 percentage points for Claude Haiku 4\.5\. This indicates that different LLM backbones rely on different documentation information\. A similar variation is observed across agent paradigms\. As shown in Fig\.[1a](https://arxiv.org/html/2608.10037#S2.F1.sf1)and Fig\.[1f](https://arxiv.org/html/2608.10037#S2.F1.sf6), replacing ReAct with Multi\-Agent reverses the effects of several fields, includingfunctionality description\(FD\),input parameter description\(IPD\),parameter required flag\(PRF\), and RT\. Therefore, although some fields provide stable benefits or harms, the effectiveness of fields depends on the task domain, LLM backbone, and agent paradigm\.

Findings\.Different information fields in tool documentation have different impacts on task success rate of LLM agents across task domains, LLM backbones, and agent paradigms\.

### II\-DEmpirical Insights

The empirical study provides two insights for tool documentation optimization\. First, no fixed documentation convention can consistently generalize across different LLM agent settings, since the effectiveness of information fields may vary across task domains, LLM backbones, and agent paradigms\. Second, an effective tool documentation optimizer should adaptively adjust the information field composition based on the LLM agent setting according to different execution feedback\.

![Refer to caption](https://arxiv.org/html/2608.10037v1/x1.png)\(a\)Data Analysis\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x2.png)\(b\)Email\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x3.png)\(c\)Calendar\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x4.png)\(d\)Data Analysis\+GLM\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x5.png)\(e\)Data Analysis\+Claude\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x6.png)\(f\)Data Analysis\+GPT\+Multi\-Agent

Figure 1:Impact of Information Fields on Task Success Rate across Task Domains, LLM Backbones, and Agent Paradigms

## IIIMethodology

Motivated by the empirical insights in Sec\.[II](https://arxiv.org/html/2608.10037#S2), we proposeDocsChisel, an adaptive framework for optimizing tool documentation for LLM agents\. Given a target LLM agent, a set of tools with their original documentation, and a set of queries with execution ground truth,DocsChiselgenerates optimized tool documentation for each tool under the target agent setting\. The key idea is to perform tool documentation optimization at the information\-field levelfor each tool under the guidance of failed execution traces of the target agent\.

### III\-AApproach Overview

![Refer to caption](https://arxiv.org/html/2608.10037v1/x7.png)Figure 2:Approach Overview ofDocsChiselFig\.[2](https://arxiv.org/html/2608.10037#S3.F2)shows the approach overview\. First,DocsChiselperforms domain\-aware execution profiling with the original tool documentation \(Sec\.[III\-B](https://arxiv.org/html/2608.10037#S3.SS2)\)\. In this stage,when the target agent is equipped with tools from multiple task domains,DocsChiselorganizes the tools and queries by task domain\. Within each domain, it associates each tool with the queries whose execution requires that tool, and partitions these tool\-associated queries and their execution ground truth into optimization and validation sets\. It then executes the target agent to collect traces on each tool’s optimization set and obtain its initial task success rate on the corresponding validation set\.

Then,DocsChiselenters an iterative optimization loop for each tool in the current task domain\. At each iteration,DocsChiselperforms memory\-guided documentation optimization for the target tool using the queries associated with that tool in the optimization set \(Sec\.[III\-C](https://arxiv.org/html/2608.10037#S3.SS3)\)\. Given the original tool documentation, the latest failed execution traces collected in the optimization set,the field optimization memories of the current domain, and the information field set,DocsChiseluses an LLM\-based exploration process to generate new tool documentation, including adding missing information fields, removing distracting or redundant fields, and refining ambiguous or incomplete fields\. Each generated tool documentation is then evaluated through agent re\-execution to update the feedback \(Sec\.[III\-D](https://arxiv.org/html/2608.10037#S3.SS4)\)\.DocsChiselfirst executes the target agent with the generated tool documentation on the optimization set to obtain new failed traces, and updates the field optimization memories for the next round of tool documentation generation\. Then, it evaluates the generated tool documentation on its validation set to update the candidate documentation set\. Once the maximum number of optimization iterations for the target tool is reached,DocsChiselselects the best documentation from its candidate set \(Sec\.[III\-E](https://arxiv.org/html/2608.10037#S3.SS5)\)\. It then repeats the optimization process for the next tool until all tools in the current domain have been processed\.

### III\-BDomain\-Aware Execution Profiling

Our empirical study shows that different task domains rely on different information fields in tool documentation\. Meanwhile, a user query may require the agent to invoke multiple tools within the same domain to complete the task, making the effectiveness of one tool’s documentation related to other semantically relevant tools\. Therefore, we first organize tools and queries by task domain, so that subsequent documentation optimization can be performed within each domain and exploit the domain\-specific information field optimization experience\.

For task domaincc, let𝒯c\\mathcal\{T\}\_\{c\}denote the tools in this domain and let𝒟c0=\{dt0∣t∈𝒯c\}\\mathcal\{D\}^\{0\}\_\{c\}=\\\{d^\{0\}\_\{t\}\\mid t\\in\\mathcal\{T\}\_\{c\}\\\}denote the tool documentation set composed of original documentationdt0d^\{0\}\_\{t\}for each tooltt\.Let𝒬c,t\\mathcal\{Q\}\_\{c,t\}denote the domain\-associated queries whose ground\-truth executions involve tooltt\. A query involving multiple tools belongs to multiple tool\-specific query sets\. We partition each𝒬c,t\\mathcal\{Q\}\_\{c,t\}into an optimization set𝒬c,to​p​t\\mathcal\{Q\}^\{opt\}\_\{c,t\}and a validation set𝒬c,tv​a​l\\mathcal\{Q\}^\{val\}\_\{c,t\}\.We further denote the collections of tool\-specific optimization and validation query sets in domainccas𝒬co​p​t=\{𝒬c,to​p​t∣t∈𝒯c\}\\mathcal\{Q\}^\{opt\}\_\{c\}=\\\{\\mathcal\{Q\}^\{opt\}\_\{c,t\}\\mid t\\in\\mathcal\{T\}\_\{c\}\\\}and𝒬cv​a​l=\{𝒬c,tv​a​l∣t∈𝒯c\}\\mathcal\{Q\}^\{val\}\_\{c\}=\\\{\\mathcal\{Q\}^\{val\}\_\{c,t\}\\mid t\\in\\mathcal\{T\}\_\{c\}\\\}, respectively\.

Then, we execute the target LLM agent on both sets using theoriginal tooldocumentation𝒟c0\\mathcal\{D\}^\{0\}\_\{c\}\. For queries in𝒬c,to​p​t\\mathcal\{Q\}^\{opt\}\_\{c,t\}, we collect execution traces, which include the user query, the reasoning steps, invoked tools, tool inputs, tool outputs, and available error messages\.Letℱc0=\{ℱc,t0∣t∈𝒯c\}\\mathcal\{F\}\_\{c\}^\{0\}=\\\{\\mathcal\{F\}\_\{c,t\}^\{0\}\\mid t\\in\\mathcal\{T\}\_\{c\}\\\}and𝒮c0=\{𝒮c,t0∣t∈𝒯c\}\\mathcal\{S\}\_\{c\}^\{0\}=\\\{\\mathcal\{S\}\_\{c,t\}^\{0\}\\mid t\\in\\mathcal\{T\}\_\{c\}\\\}denote the failed and successful execution trace collections obtained using the original tool documentation𝒟c0\\mathcal\{D\}^\{0\}\_\{c\}, whereℱc,t0\\mathcal\{F\}\_\{c,t\}^\{0\}and𝒮c,t0\\mathcal\{S\}\_\{c,t\}^\{0\}represent the failed and successful traces involving tooltt, respectively\. The failed traces are used as evidence for optimizing tool documentation, while the successful traces are retained for regression checking\. For queries in validation sets, we record the initial task success rateTSc0=\{TSc,t0∣t∈𝒯c\}\\text\{TS\}^\{0\}\_\{c\}=\\\{\\text\{TS\}^\{0\}\_\{c,t\}\\mid t\\in\\mathcal\{T\}\_\{c\}\\\}of the target LLM agent as a reference before documentation optimization\.Finally, the output of this stage is a profiling result⟨𝒯c,𝒟c0,𝒬co​p​t,𝒬cv​a​l,ℱc0,𝒮c0,TSc0⟩\\langle\\mathcal\{T\}\_\{c\},\\mathcal\{D\}^\{0\}\_\{c\},\\mathcal\{Q\}^\{opt\}\_\{c\},\\mathcal\{Q\}^\{val\}\_\{c\},\\mathcal\{F\}^\{0\}\_\{c\},\\mathcal\{S\}^\{0\}\_\{c\},\\text\{TS\}^\{0\}\_\{c\}\\ranglefor each task domain\.

### III\-CMemory\-Guided Documentation Optimization

![Refer to caption](https://arxiv.org/html/2608.10037v1/x8.png)Figure 3:Memory\-Guided Documentation OptimizationWithin each task domaincc,DocsChiseloptimizes the original tool documentationdt0d^\{0\}\_\{t\}of each toolt∈𝒯ct\\in\\mathcal\{T\}\_\{c\}iteratively\. At iterationnn, instead of directly asking an LLM to rewrite the current documentation\[[58](https://arxiv.org/html/2608.10037#bib.bib56),[9](https://arxiv.org/html/2608.10037#bib.bib61)\],DocsChiseldecomposes the optimization process into three LLM\-assisted steps, i\.e\.,failed trace diagnosis,field operation planning, andoptimized documentation generation, as illustrated in Fig\.[3](https://arxiv.org/html/2608.10037#S3.F3)\. The diagnosis step identifies why the LLM agent fails under the current documentation\. The planning step maps the diagnosed issues to possible field\-level operations\. The generation step applies the planned operations to produce new tool documentationdtnd\_\{t\}^\{n\}\.

Failed Trace Diagnosis\.At iterationnn,DocsChiselfirst diagnoses the failed execution tracesℱc,tn−1\\mathcal\{F\}\_\{c,t\}^\{n\-1\}collected fromoptimization set𝒬c,to​p​t\\mathcal\{Q\}^\{opt\}\_\{c,t\}under the latest documentationdtn−1d\_\{t\}^\{n\-1\}\. Since execution failures may be caused by factors beyond tool documentation, we prompt an LLM to inspect each failed trace together with the current documentationdtn−1d\_\{t\}^\{n\-1\}, the ground truth, and the diagnosis memoryℳcdiag\\mathcal\{M\}\_\{c\}^\{\\text\{diag\}\}of the current task domain\.

The diagnosis memory stores domain\-specific failure patterns observed in previous iterations, including frequently occurring agent errors and their potentially related information fields\. By referring to such memory, the LLMcan reuse diagnosis experience accumulated from previous iterations, rather than analyzing each failed trace in isolation\. Fig\.[4](https://arxiv.org/html/2608.10037#S3.F4)shows the template of the diagnosis prompt for the LLM\. Specifically, the LLM is required to summarize the failure evidence, identify the mismatch between agent behaviors and expected tool invocations, and infer the potentially responsible information fields from those summarized in Table[II](https://arxiv.org/html/2608.10037#S2.T2)for failure cases\.

Formally, for each failed tracef∈ℱc,tn−1f\\in\\mathcal\{F\}\_\{c,t\}^\{n\-1\}, we obtain a diagnosis recordrf=⟨ef,bf,ℋf⟩r\_\{f\}=\\langle e\_\{f\},b\_\{f\},\\mathcal\{H\}\_\{f\}\\rangle, whereefe\_\{f\}denotes the summarized failure evidence,bfb\_\{f\}denotes the agent behavior mismatch with respect to the ground truth, andℋf\\mathcal\{H\}\_\{f\}denotes the set of potentially related information fields\. The diagnosis records of all failed traces form the setℛtn=\{rf∣f∈ℱc,tn−1\}\\mathcal\{R\}\_\{t\}^\{n\}=\\\{r\_\{f\}\\mid f\\in\\mathcal\{F\}\_\{c,t\}^\{n\-1\}\\\}, which is then passed to the field operation planning step\.

![Refer to caption](https://arxiv.org/html/2608.10037v1/x9.png)Figure 4:Prompt Template for Failed Trace DiagnosisField Operation Planning\.After obtaining the diagnosis setℛtn\\mathcal\{R\}\_\{t\}^\{n\},DocsChiselfurther plans how to revise the tool documentation at the information\-field level\. Givenℛtn\\mathcal\{R\}\_\{t\}^\{n\}, the original documentationdt0d\_\{t\}^\{0\}, and the planning memoryℳcplan\\mathcal\{M\}\_\{c\}^\{\\text\{plan\}\}of the current task domain, we prompt an LLM to map the diagnosed issues to concrete information field operations\. Here,dt0d\_\{t\}^\{0\}serves as the semantic anchor for planning operations, preventing the optimization process from drifting away from the original tool documentation semantics\.

The planning memory stores domain\-specific operation experience observed in previous iterations, including which field operations are useful for addressing certain failure patterns\. For example, missing invocation conditions can be mapped to adding invocation constraints, distracting usage examples or implementation snippets can be mapped to removing redundant fields, and confusion between similar tools can be mapped to refining functionality descriptions or usage guidance\. By referring to such memory,DocsChiselavoids planning field operations from each failed trace in isolation, and instead exploits historical optimization experience from related tools in the same domain\. Fig\.[5](https://arxiv.org/html/2608.10037#S3.F5)shows the template of the planning prompt for the LLM\. Specifically, the LLM is required to examine each diagnosis record inℛtn\\mathcal\{R\}\_\{t\}^\{n\}, determine whether the diagnosed issue requires adding, removing, or refining an information field, and provide the rationale for the operation\.

Formally, for each diagnosis recordrf∈ℛtnr\_\{f\}\\in\\mathcal\{R\}\_\{t\}^\{n\}, we obtain an information\-field operation planpf=\{⟨of,hf,gf⟩∣hf∈ℋfp\_\{f\}=\\\{\\langle o\_\{f\},h\_\{f\},g\_\{f\}\\rangle\\mid h\_\{f\}\\in\\mathcal\{H\}\_\{f\}\}, whereof∈\{Add,Remove,Refine\}o\_\{f\}\\in\\\{\\textsc\{Add\},\\textsc\{Remove\},\\textsc\{Refine\}\\\}denotes the planned operation,hf∈ℋfh\_\{f\}\\in\\mathcal\{H\}\_\{f\}denotes the target information field, andgfg\_\{f\}denotes the reason for applying this operation\. Since different failed traces may lead to duplicated or conflicting operations, we merge trace\-level plans into a unified operation plan𝒫tn\\mathcal\{P\}\_\{t\}^\{n\}by removing duplicated operations and resolving conflicts according to their supporting diagnosis records, the planning memory, and the original documentation\. The resulting plan𝒫tn\\mathcal\{P\}\_\{t\}^\{n\}specifies which information fields should be added, removed, or refined in the next documentation generation step\.

![Refer to caption](https://arxiv.org/html/2608.10037v1/x10.png)Figure 5:Prompt Template for Field Operation PlanningOptimized Documentation Generation\.Given the unified operation plan𝒫tn\\mathcal\{P\}\_\{t\}^\{n\},DocsChiselgenerates new tool documentation for tooltt, using the original documentationdt0d\_\{t\}^\{0\}as the generation anchor to preserve the original tool semantics and reduce the risk of information drift during repeated rewriting\.

To guide the generation process,DocsChiselfurther incorporates the generation memoryℳcgen\\mathcal\{M\}\_\{c\}^\{\\text\{gen\}\}of the current task domain\. The generation memory stores domain\-specific editing experience observed in previous iterations, includingfield expressions that are effective for mitigating certain failure patterns, generation constraints, and edits that may introduce regression errors\. Fig\.[6](https://arxiv.org/html/2608.10037#S3.F6)shows the template of the generation prompt for the LLM\. Specifically, the LLM is required to apply the operations in𝒫tn\\mathcal\{P\}\_\{t\}^\{n\}todt0d\_\{t\}^\{0\}by adding missing fields, removing distracting or redundant fields, and refining ambiguous or incomplete fields, while preserving unchanged information that is not targeted by the operation plan\.

![Refer to caption](https://arxiv.org/html/2608.10037v1/x11.png)Figure 6:Prompt Template for Documentation Generation
### III\-DCandidate Evaluation and Feedback Update

After generating the candidate documentationdtnd\_\{t\}^\{n\}, we evaluate whether it improves the task success rate of the target LLM agent through agent re\-execution, and update the optimization feedback for subsequent iterations\. Since the optimized documentation may improve some failed queries while harming previously successful ones, we consider both validation performance and regression risk during the evaluation\.

Regression\-Aware Evaluation\.For the target tooltt, we replace its original documentationdt0d\_\{t\}^\{0\}with the new documentationdtnd\_\{t\}^\{n\}while keeping the documentation of other tools unchanged\. Then, we execute the LLM agent on𝒬c,tv​a​l\\mathcal\{Q\}^\{val\}\_\{c,t\}and compute the task successTSc,tn\\text\{TS\}\_\{c,t\}^\{n\}\. In addition, we check whetherdtnd\_\{t\}^\{n\}introduces regression errors on previously successful optimization queries by re\-executing the LLM agent on𝒬c,to​p​t\\mathcal\{Q\}^\{opt\}\_\{c,t\}to obtain the failed trace setℱc,tn\\mathcal\{F\}\_\{c,t\}^\{n\}and the successful trace set𝒮c,tn\\mathcal\{S\}\_\{c,t\}^\{n\}\. Based on these results, we obtain the regression set𝒢c,tn\\mathcal\{G\}\_\{c,t\}^\{n\}, which contains the failed traces of queries that were successfully completed under the original documentationdt0d\_\{t\}^\{0\}but failed underdtnd\_\{t\}^\{n\}\.

Candidate and Memory Update\.We compare the newly generated documentationdtnd\_\{t\}^\{n\}with the original documentationdt0d\_\{t\}^\{0\}using the validation task success rateTSc,t0\\text\{TS\}\_\{c,t\}^\{0\}obtained during execution profiling\. Only whenTSc,tn≥TSc,t0\\text\{TS\}\_\{c,t\}^\{n\}\\geq\\text\{TS\}\_\{c,t\}^\{0\}, we add the new tool documentationdtnd\_\{t\}^\{n\}to the candidate documentation set𝒟ct\\mathcal\{D\}^\{t\}\_\{c\}\.

![Refer to caption](https://arxiv.org/html/2608.10037v1/x12.png)Figure 7:Prompt Template for Updating the MemoryMeanwhile, the failed tracesℱc,tn\\mathcal\{F\}\_\{c,t\}^\{n\}collected on𝒬c,to​p​t\\mathcal\{Q\}^\{opt\}\_\{c,t\}are used as the input for failed trace diagnosis in the next iteration\. We further update the domain\-level memory based on the evaluation results\. Specifically, we prompt an LLM to analyze the new documentationdtnd\_\{t\}^\{n\}, the regression set𝒢c,tn\\mathcal\{G\}\_\{c,t\}^\{n\}and the validation resultTSc,tn\\text\{TS\}\_\{c,t\}^\{n\}\. Fig\.[7](https://arxiv.org/html/2608.10037#S3.F7)shows the prompt template for the LLM\. Specifically, we require the LLM to summarize the newly observed failure patterns, effective field operations and editing experience during this optimization iteration\. Then, we update theℳcdiag\\mathcal\{M\}^\{\\text\{diag\}\}\_\{c\},ℳcplan\\mathcal\{M\}^\{\\text\{plan\}\}\_\{c\}andℳcgen\\mathcal\{M\}^\{\\text\{gen\}\}\_\{c\}, respectively\. In this way, the updated traces and memories provide feedback for the next iteration, forming a closed optimization loop\.

### III\-EFinal Documentation Selection

When optimization for toolttreaches the maximum number of iterations, we select the candidate with the highest validation task success rate from𝒟ct\\mathcal\{D\}\_\{c\}^\{t\}, breaking ties by choosing the shortest documentation to reduce execution context cost\. We then proceed to the next tool until all tools in𝒯c\\mathcal\{T\}\_\{c\}are processed\.

## IVEvaluation

We implement a prototype ofDocsChiselwith2,687lines of Python code\. To evaluate the effectiveness and efficiency ofDocsChisel, we design the following research questions\.

- •RQ3 Effectiveness Evaluation\.What is the effectiveness ofDocsChiselin improving task success rate of LLM agents?
- •RQ4 Efficiency Evaluation\.What is the efficiency and optimization overhead ofDocsChisel?
- •RQ5 Sensitivity Analysis\.How do optimization models and iteration budgets affect the effectiveness ofDocsChisel?
- •RQ6 Ablation Study\.What is the contribution of our memory mechanism to the effectiveness ofDocsChisel?

### IV\-AEvaluation Setup

Dataset\.We select two tool\-use benchmarks,i\.e\.,WorkBench\[[48](https://arxiv.org/html/2608.10037#bib.bib52)\]and API\-Bank\[[27](https://arxiv.org/html/2608.10037#bib.bib46)\]\. In addition to the five task domains from WorkBench used inRQ2, API\-Bank provides tools from four additional domains,i\.e\.,finance management, health management, information retrieval, and smart home management \(SHM\)\. Overall, our evaluation dataset covers 9 task domains, involving74tools and2,072user queries\. We split the user queries into optimization, validation, and test sets with a ratio of 5:1:4\. The optimization and validation sets are used during documentation optimization, while the test set is held out throughout the optimization process and used only for the final evaluation of the optimized tool documentation\. To ensure a fair and representative evaluation, we perform a stratified split over task domains and tools, so that each subset preserves the overall distribution of tool\-use scenarios as much as possible\. After splitting, the optimization, validation, and test sets contain1,036,207, and829user queries, respectively\. The detailed statistics of the evaluation dataset across task domains and tools are available at\[[1](https://arxiv.org/html/2608.10037#bib.bib1)\]due to space limitations\.

Metrics\.Beyond theTSmetric used in our empirical study, we also usetool invocation correctness\(TC\) to measure whether the LLM agent invokes the correct tools\. Specifically, given the test suite containing tool\-use queries𝒬\\mathcal\{Q\}, for each queryqi∈𝒬\{q\_\{i\}\}\\in\\mathcal\{Q\}, we denote the ground\-truth tool set asTqiT\_\{q\_\{i\}\}and the tool set predicted by the LLM agent asT^qi\\hat\{T\}\_\{q\_\{i\}\}\. A query is regarded as tool\-correct only when the predicted tool set exactly matches the ground\-truth set,i\.e\.,Tqi=T^qiT\_\{q\_\{i\}\}=\\hat\{T\}\_\{q\_\{i\}\}\. We compute tool invocation correctness asTC=1\|𝒬\|​∑i=1\|𝒬\|𝕀​\(Tqi=T^qi\)\\text\{TC\}=\\frac\{1\}\{\|\\mathcal\{Q\}\|\}\\sum\_\{i=1\}^\{\|\\mathcal\{Q\}\|\}\\mathbb\{I\}\(T\_\{q\_\{i\}\}=\\hat\{T\}\_\{q\_\{i\}\}\), where𝕀​\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function\. Together,TCandTScapture the effectiveness of tool documentation on both tool invocation behavior and task completion performance of LLM agents\.

Baseline\.We compareDocsChiselwith two tool documentation optimization approaches,i\.e\.,EasyTool\[[58](https://arxiv.org/html/2608.10037#bib.bib56)\]andDRAFT\[[42](https://arxiv.org/html/2608.10037#bib.bib57)\]\.EasyTooluses an LLM and a predefined template to convert documentation into concise, structured tool\-use instructions\.DRAFTuses LLMs to iteratively refine tool documentation based on feedback obtained by generating tool inputs and executing the tools\. We use their official artifacts and adapt only the input interfaces to our dataset\.

RQ Setup\.ForRQ3andRQ4, we applyDocsChiselwith Claude Haiku 4\.5 as the optimization model to optimize the original tool documentation in our dataset\.We set the maximum number of iterations to 5 by default to balance optimization effectiveness and LLM token overhead\.ForRQ3, we evaluate the effectiveness ofDocsChiselby comparing it with using the original tool documentation and using the tool documentation optimized byEasyToolandDRAFT, respectively\. To understand whether the improvement generalizes across different agent settings, we conduct comparisons from three perspectives introduced inRQ2,i\.e\.,task domains, LLM backbones, and agent paradigms, respectively\.

ForRQ4, we report the additional token cost introduced by the optimized documentation during agent execution, compared with the original documentation\. We also measure the optimization overhead ofDocsChisel, including the time cost and token consumption required to generate optimized tool documentation, and compare them with the baselines\.

ForRQ5, we examine how different optimization models and iteration budgets affectDocsChiselin the data analysis domain\. We vary the LLM used for documentation optimization,i\.e\.,GPT\-4o and GLM\-5, and the maximum number of optimization iterations, while keeping the original tool documentation, query set, and target agent unchanged\. The target agent is a GPT\-4o\-based ReAct agent\. We report the averageTCandTSon the test set using the optimized documentation generated under each setting\.

ForRQ6, we conduct an ablation study to evaluate the contribution of our memory mechanism\. We construct a variant ofDocsChisel, denoted asDocsChisel\-NoMem, by disabling the memory mechanism while keeping the remaining optimization pipeline unchanged\. We then compare the tool documentation in the data analysis domain optimized byDocsChiselandDocsChisel\-NoMem, and report the averageTCandTSof the GPT\-4o\-based ReAct agent, analyzing whether the memory mechanism helpsDocsChiselproduce more effective tool documentation for the target LLM agent\.

Environment\.We conduct all the experiments on Ubuntu 20\.04\.4 LTS servers with 4 NVIDIA GeForce RTX 3090 GPUs, Intel\(R\) Xeon\(R\) Silver 4310 @ 2\.10GHz and 128GB memory\.

![Refer to caption](https://arxiv.org/html/2608.10037v1/x13.png)\(a\)Data Analysis\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x14.png)\(b\)Email\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x15.png)\(c\)Calendar\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x16.png)\(d\)CRM\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x17.png)\(e\)Health\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x18.png)\(f\)Information\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x19.png)\(g\)SHM\+GPT\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x20.png)\(h\)Data Analysis\+GLM\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x21.png)\(i\)Data Analysis\+Claude\+ReAct
![Refer to caption](https://arxiv.org/html/2608.10037v1/x22.png)\(j\)Data Analysis\+GPT\+Multi\-Agent

Figure 8:Effectiveness Comparison across Task Domains, LLM Backbones, and Agent Paradigms
### IV\-BEffectiveness Evaluation \(RQ3\)

Overall Results\.Fig\.[8](https://arxiv.org/html/2608.10037#S4.F8)comparesDocsChisel\(DC\) with the original documentation \(OG\),EasyTool\(ET\), andDRAFT\(DRT\) across task domains, LLM backbones, and agent paradigms\. We independently repeat the optimization process three times\. For each optimized result, we conduct five independent evaluation runson the test setto account for execution randomness\. Hence, each boxplot summarizes the results from3×5=153\\times 5=15runs\. Overall,DocsChiselachieves the highest averageTCandTSin all evaluated settings\. Compared with the original documentation,DocsChiselimprovesTCby34\.69%andTSby95\.89%on average\. Compared with the two baseline approaches, it achieves an averageTCimprovement of30\.83%andTSimprovement of75\.15%\. Even against the strongest baseline in each setting,DocsChiselimprovesTCandTSby26\.23%and46\.11%on average, respectively\.

With respect to task domains, Fig\.[8a](https://arxiv.org/html/2608.10037#S4.F8.sf1)\-[8g](https://arxiv.org/html/2608.10037#S4.F8.sf7)show thatDocsChiselconsistently achieves the highest averageTCandTSin the seven domains presented in the paper\. The remaining two domains,i\.e\.,project management and finance management, are reported on our website\[[1](https://arxiv.org/html/2608.10037#bib.bib1)\]due to space limitations\. Across all nine evaluated domains,DocsChiselimprovesTCandTSby46\.27%and194\.22%over the original documentation, respectively, and by27\.34%and42\.36%over the strongest documentation optimization baseline on average\. The gains are particularly evident in domains such as email management, customer relationship management, health management, and smart home management, where the original and baseline documentation often yields low task success rate despite moderate tool invocation correctness\.

With respect to LLM backbones, Fig\.[8a](https://arxiv.org/html/2608.10037#S4.F8.sf1), Fig\.[8h](https://arxiv.org/html/2608.10037#S4.F8.sf8), and Fig\.[8i](https://arxiv.org/html/2608.10037#S4.F8.sf9)show thatDocsChiselremains best\-performing for all three backbones\. Compared with the strongest baseline, it improvesTCby3\.45%andTSby60\.83%, on average, across GPT\-4o, GLM\-5, and Claude Haiku 4\.5\.

With respect to agent paradigms, Fig\.[8a](https://arxiv.org/html/2608.10037#S4.F8.sf1)and Fig\.[8j](https://arxiv.org/html/2608.10037#S4.F8.sf10)compare ReAct and Multi\-Agent under the same task domain and GPT\-4o backbone\.DocsChiselachieves the highestTCandTSunder both paradigms\. Compared with the strongest baseline, it improvesTCby3\.44%and3\.34%, andTSby63\.61%and58\.35%under ReAct and Multi\-Agent, respectively\.

The results further show thatDocsChiselgenerally achieves higher lower\-end performance and more compact distributions across repeated runs\. Compared withEasyToolandDRAFT,DocsChiselreduces the average interquartile range ofTCby41\.56%and52\.93%, respectively, and that ofTSby36\.71%and44\.07%\. These results indicate thatDocsChiselintroduces stable improvements across different agent settings\.

Breakdown Analysis\.EasyToolimproves some settings by transforming documentation into concise and structured instructions\. However, its fixed template applies the same information fields across agent settings\. It may remove fields useful to a particular agent or fail to introduce fields absent from the original documentation, resulting in limited improvements\.

DRAFTuses feedback obtained by generating tool inputs and executing tools to iteratively refine tool documentation\. Although such feedback helps correct tool descriptions, it mainly reflects isolated tool executions rather than failures occurring during end\-to\-end agent task solving\. Moreover, it does not explicitly adapt the composition of information fields\. Consequently, documentation\-related failures observed in complete agent traces may remain unresolved\.

In contrast,DocsChiseldiagnoses failed agent execution traces and performs field\-level addition, removal, and refinement according to the target agent setting, allowing for better alignment of tool documentation with the information requirements of the target agent, explaining its consistent improvements in bothTCandTS\. Nevertheless,DocsChiseldoes not achieve perfect results\. Some failures originate from factors beyond tool documentation, such as incorrect task decomposition, stochastic LLM reasoning, or incomplete user queries\. Therefore, documentation optimization can substantially improve agent effectiveness, but cannot eliminate failures caused by other components of the agent execution process\.

Summary\.DocsChiselconsistently outperformsEasyToolandDRAFTacross task domains, LLM backbones, and agent paradigms, improving tool invocation correctness by30\.83%and task success rate by75\.15%, on average\.

### IV\-CEfficiency Evaluation \(RQ4\)

TABLE III:Results of Efficiency EvaluationApproachDoc LengthToken CostTime Cost \(min\)Original152\.58––EasyTool82\.68237\.810\.23DRAFT189\.564,784\.392\.26DocsChisel189\.234,480\.8112\.65Table[III](https://arxiv.org/html/2608.10037#S4.T3)reports the average documentation length and optimization overhead per tool\.EasyToolproduces the shortest documentation because it primarily compresses the original content\. In contrast,DRAFTandDocsChiselgenerate slightly longer documentation by incorporating additional information\. The documentation produced byDocsChiselcontains 189\.23 tokens on average, which is24\.02%longer than the original documentation but comparable toDRAFT\. Thus,DocsChiselintroduces only a limited additional context cost\.

For optimization overhead,DocsChiselconsumes6\.35%fewer tokens thanDRAFTfor each tool during optimization\. However, it requires 12\.65 minutes on average, longer thanEasyToolandDRAFT, becauseDocsChiselrepeatedly diagnoses failed traces, generates candidate documentation, and evaluates candidates through agent re\-execution\. Overall,DocsChiseltrades additional offline optimization time for substantially improved effectiveness, while maintaining comparable documentation length and token consumption toDRAFT\.

Summary\.DocsChiselincurs a higher optimization\-time cost, requiring 12\.65 minutes per tool on average, while consuming6\.35%fewer optimization tokens than DRAFT and producing documentation of comparable length\.

### IV\-DSensitivity Analysis \(RQ5\)

TABLE IV:Results of Model SensitivityMetricsGPT\-4oGLM\-5Claude Haiku 4\.5TC47\.50%73\.75%93\.75%TS21\.25%35\.63%56\.25%![Refer to caption](https://arxiv.org/html/2608.10037v1/x23.png)Figure 9:Results of Iteration SensitivityModel Sensitivity\.Table[IV](https://arxiv.org/html/2608.10037#S4.T4)reports the effectiveness ofDocsChiselusing different optimization models while keeping the target agent and evaluation setting unchanged\. Claude Haiku 4\.5 achieves the best performance, improvingTCandTSover GLM\-5 by27\.12%and57\.87%, respectively, and over GPT\-4o by97\.37%and164\.71%\. GLM\-5 also outperforms GPT\-4o, with relative gains of55\.26%inTCand67\.67%inTS\. These results indicate that the quality of optimized documentation is sensitive to the optimization model\. Documentation optimization requires the model to diagnose failed traces, identify field\-level deficiencies, and generate appropriate revisions\. Models with stronger instruction\-following and reasoning capabilities can produce more effective documentation\.

Iteration Sensitivity\.Fig\.[9](https://arxiv.org/html/2608.10037#S4.F9)shows that increasing the number of optimization iterations improves bothTCandTS\. The gains are most pronounced in the early rounds\. Compared with one iteration, five iterations improveTSby80\.00%andTCby14\.50%\. This indicates that iterative execution feedback progressively helpsDocsChiselidentify and address documentation deficiencies\. After five iterations, however, the marginal gains become small, suggesting that the optimization process has largely converged\. We therefore set the default iteration budget to five to avoid large optimization overhead\.

Summary\.Stronger optimization models generally yield better documentation quality, while iterative refinement substantially improves optimization effectiveness\. The budget of five iterations achieves a favorable trade\-off\.

### IV\-EAblation Study \(RQ6\)

![Refer to caption](https://arxiv.org/html/2608.10037v1/x24.png)Figure 10:Results of Ablation StudyFig\.[10](https://arxiv.org/html/2608.10037#S4.F10)comparesDocsChiselwithDocsChisel\-NoMem over five optimization iterations\. In the first two iterations, the two variants achieve similar performance, indicating that both can improve documentation using the feedback from the current iteration\. However, their performance diverges as optimization proceeds\. At the fifth iteration,DocsChisel\-NoMem reducesTCandTSby13\.64%and76\.47%compared withDocsChisel, respectively\. Moreover, the relativeTSimprovement betweenDocsChiselandDocsChisel\-NoMem increases from4\.17%at Iteration\-1 to76\.47%at Iteration\-5, indicating that the benefit of memory becomes more pronounced as optimization experience accumulates\. Without memory, both metrics largely plateau after the second iteration\. In contrast,DocsChiselcontinues to improve by reusing previously observed documentation deficiencies, effective field operations, and editing experience\. Such experience guides subsequent diagnosis, field planning, and candidate generation, preventing the optimizer from repeatedly exploring ineffective revisions\.

Summary\.The memory mechanism is critical to iterative documentation optimization, improving the finalTCandTS, on average, by13\.64%and76\.47%, respectively\.

## VThreats to Validity

First, the selection of agent paradigms, LLM backbones, and datasets may limit the generalizability of our findings\. To mitigate this threat, we evaluateDocsChiselusing two agent paradigms with three LLM backbones across nine task domains\.

Second, our evaluation metrics may not capture all aspects of agent tool\-use effectiveness\. To reduce this threat, we report complementary metrics that evaluate both tool invocation correctness and final task success rate\. Moreover, identifying information fields and constructing documentation variants involve manual inspection and correction, which may introduce subjective bias\. To mitigate this threat, multiple authors independently conducted the analysis and resolved disagreements through discussion, achieving a Cohen’s kappa coefficient of 0\.862, with strong inter\-rater agreement\.

Finally, the inherent randomness of LLMs and the instability of agent execution pose additional threats to validity\. To mitigate these threats, we keep the prompts and execution environments unchanged within each comparison, use the official replication artifacts of all baselines, and conduct three independent optimization runs, each followed by five evaluation runs\. Thus, each evaluated setting is assessed over3×5=153\\times 5=15runs\. Across our evaluation, we apply the Mann\-Whitney U test\[[31](https://arxiv.org/html/2608.10037#bib.bib65)\], and the improvements ofDocsChiselremain statistically significant after Holm correction\[[16](https://arxiv.org/html/2608.10037#bib.bib64)\]\(padj<0\.05\\mathrm\{p\}\_\{\\mathrm\{adj\}\}<0\.05\)\.

## VIRelated Work

### VI\-ALLM Agent Ecosystems

Recent advances in LLM agents and agent ecosystems have substantially improved the capability of LLMs to interact with external tools\[[25](https://arxiv.org/html/2608.10037#bib.bib20),[7](https://arxiv.org/html/2608.10037#bib.bib13),[17](https://arxiv.org/html/2608.10037#bib.bib29),[15](https://arxiv.org/html/2608.10037#bib.bib31),[52](https://arxiv.org/html/2608.10037#bib.bib32),[3](https://arxiv.org/html/2608.10037#bib.bib25)\]\. Specifically, ReAct\[[56](https://arxiv.org/html/2608.10037#bib.bib21)\]introduces the reasoning\-and\-acting paradigm for iterative tool invocation, while Toolformer\[[44](https://arxiv.org/html/2608.10037#bib.bib9)\]demonstrates that LLMs can learn tool\-use behaviors through self\-supervised training\. AutoGen\[[51](https://arxiv.org/html/2608.10037#bib.bib22)\]provides a unified framework for multi\-agent orchestration and workflow automation\. Subsequent works\[[38](https://arxiv.org/html/2608.10037#bib.bib10),[27](https://arxiv.org/html/2608.10037#bib.bib46),[40](https://arxiv.org/html/2608.10037#bib.bib11)\]further improve large\-scale tool\-use capabilities of LLM agents\. Representative agent systems such as Claude Code\[[2](https://arxiv.org/html/2608.10037#bib.bib27)\], OpenAI Codex\[[36](https://arxiv.org/html/2608.10037#bib.bib28)\], and SWE\-agent\[[54](https://arxiv.org/html/2608.10037#bib.bib8)\]have demonstrated the growing applicability of tool\-use LLM agents in software engineering and other complex domains\[[28](https://arxiv.org/html/2608.10037#bib.bib33)\]\. To evaluate the reliability and execution capability of tool\-use LLM agents, recent studies\[[53](https://arxiv.org/html/2608.10037#bib.bib50),[55](https://arxiv.org/html/2608.10037#bib.bib44),[48](https://arxiv.org/html/2608.10037#bib.bib52),[8](https://arxiv.org/html/2608.10037#bib.bib45),[45](https://arxiv.org/html/2608.10037#bib.bib48),[57](https://arxiv.org/html/2608.10037#bib.bib53),[23](https://arxiv.org/html/2608.10037#bib.bib43),[21](https://arxiv.org/html/2608.10037#bib.bib47),[50](https://arxiv.org/html/2608.10037#bib.bib49),[22](https://arxiv.org/html/2608.10037#bib.bib51)\]propose diverse benchmarks that include tools with documentation\.

Although these studies significantly advance tool\-use capabilities of LLM agents, they mainly focus on tool construction, retrieval, invocation, and evaluation strategies, while largely treating tool documentation as fixed input for tool grounding and execution\. In contrast, our work systematically investigates the structure and effectiveness of tool documentation itself, and studies how adaptive tool documentation optimization affects downstream task success rate across different agent settings\.

### VI\-BTool Documentation Engineering

Software engineering research has long recognized the importance of API usability, interface specifications, and documentation quality in supporting correct software usage\[[6](https://arxiv.org/html/2608.10037#bib.bib34),[5](https://arxiv.org/html/2608.10037#bib.bib40),[30](https://arxiv.org/html/2608.10037#bib.bib37),[39](https://arxiv.org/html/2608.10037#bib.bib35),[32](https://arxiv.org/html/2608.10037#bib.bib36),[49](https://arxiv.org/html/2608.10037#bib.bib39)\]\. These works focus on API documentation generation\[[33](https://arxiv.org/html/2608.10037#bib.bib41)\], API documentation optimization\[[46](https://arxiv.org/html/2608.10037#bib.bib38)\]and documentation smell detection\[[24](https://arxiv.org/html/2608.10037#bib.bib42)\]to improve documentation readability, completeness, and consistency for human developers\. With the rapid development of LLM agents, tool documentation has also attracted research attention\. Patel et al\.\[[9](https://arxiv.org/html/2608.10037#bib.bib61)\]automatically generate executable tool\-use demonstrations to reduce the manual effort of writing tool documentation, whileEasyTool\[[58](https://arxiv.org/html/2608.10037#bib.bib56)\]compresses existing tool documentation into concise instructions using predefined templates to reduce context overhead\. A few works\[[10](https://arxiv.org/html/2608.10037#bib.bib60),[42](https://arxiv.org/html/2608.10037#bib.bib57),[26](https://arxiv.org/html/2608.10037#bib.bib63)\]further leverage LLMs and feedback from tool execution to iteratively refine the tool documentation for better tool invocation correctness\.

These studies improve tool documentation quality and the tool\-use capabilities of LLM agents to some extent\. However, they mainly optimize tool documentation from an isolated, tool\-level perspective, focusing on rewriting, correction, compression, or refinement within existing information fields\. In contrast,DocsChiseldoes not treat documentation of each tool as an independent optimization target\. Instead, it considers the interactions among tools within the same task domain and transfers optimization experience derived from domain\-level and agent\-aware execution failures to improve individual tool documentation by adding, removing, or refining information fields\. In this way,DocsChiselaims not only to enhance the invocation correctness of each tool, but also to improve the overall task success of different LLM agents\.

## VIIConclusion

We conduct an empirical study of tool documentation for LLM agents, revealing the varying effectiveness of different information fields on task success rate across task domains, LLM backbones, and agent paradigms\. Our findings demonstrate that fixed tool documentation cannot consistently support different agent settings\. Thus, we proposeDocsChisel, an adaptive tool documentation optimization framework that automatically refines tool documentation based on the execution traces of LLM agents\. Large\-scale experiments have been conducted to demonstrate the effectiveness and efficiency ofDocsChisel\.

## References

- \[1\]Anonymous\(2026\)DocsChisel\(Website\)Note:Accessed on May 11, 2026External Links:[Link](https://anonymous.4open.science/r/F42D-DocsChisel)Cited by:[§II\-C](https://arxiv.org/html/2608.10037#S2.SS3.p1.4.3),[§IV\-A](https://arxiv.org/html/2608.10037#S4.SS1.p1.1),[§IV\-B](https://arxiv.org/html/2608.10037#S4.SS2.p2.1)\.
- \[2\]Anthropic\(2025\)Claude code\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://www.anthropic.com/claude-code)Cited by:[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[3\]Anthropic\(2025\)Writing effective tools for agents — with agents\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://www.anthropic.com/engineering/writing-tools-for-agents)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p5.1),[§II\-A](https://arxiv.org/html/2608.10037#S2.SS1.p2.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[4\]ByteDance\(2024\)Coze\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://www.coze.com/)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1)\.
- \[5\]C\. R\. B\. de Souza and D\. L\. M\. Bentolila\(2009\)Automatic evaluation of api usability using complexity metrics and visualizations\.InProceedings of the 31st International Conference on Software Engineering,pp\. 299–302\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[6\]U\. Dekel and J\. D\. Herbsleb\(2009\)Improving api documentation usability with knowledge pushing\.InProceedings of the IEEE 31st International Conference on Software Engineering,pp\. 320–330\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[7\]Dify\.AI\(2024\)Dify\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://dify.ai/)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[8\]Y\. Du, F\. Wei, and H\. Zhang\(2024\)AnyTool: self\-reflective, hierarchical agents for large\-scale api calls\.InProceedings of the 41st International Conference on Machine Learning,pp\. 11812–11829\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.8.1),[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[9\]W\. Fang, Y\. Zhang, K\. Qian, J\. Glass, and Y\. Zhu\(2025\)Play2prompt: zero\-shot tool instruction optimization for llm agents via tool play\.InProceedings of the Findings of the Association for Computational Linguistics,pp\. 26274–26290\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§III\-C](https://arxiv.org/html/2608.10037#S3.SS3.p1.5),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[10\]S\. Ghoshal, A\. Mittal, J\. Singh, M\. Ballesteros, W\. Sun, F\. Tu, S\. Singh, Y\. Benajiba, F\. Shah, S\. Bharadwaj,et al\.\(2026\)JTPRO: a joint tool\-prompt reflective optimization framework for language agents\.arXiv preprint arXiv:2604\.19821\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[11\]T\. Glm, A\. Zeng, B\. Xu, B\. Wang, C\. Zhang, D\. Yin, D\. Zhang, D\. Rojas, G\. Feng, H\. Zhao,et al\.\(2024\)Chatglm: a family of large language models from glm\-130b to glm\-4 all tools\.arXiv preprint arXiv:2406\.12793\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1)\.
- \[12\]Google\(2023\)Google gemini\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://gemini.google.com/)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1)\.
- \[13\]S\. Gravitas\(2023\)AutoGPT: build, deploy, and run ai agents\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://github.com/significant-gravitas/autogpt)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1)\.
- \[14\]R\. Guo, K\. Dong, X\. Gao, and K\. Das\(2026\)Learning to rewrite tool descriptions for reliable llm\-agent tool use\.arXiv preprint arXiv:2602\.20426\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p3.1)\.
- \[15\]J\. He, C\. Treude, and D\. Lo\(2025\)Llm\-based multi\-agent systems for software engineering: literature review, vision, and the road ahead\.ACM Transactions on Software Engineering and Methodology34\(5\),pp\. 1–30\.Cited by:[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[16\]S\. Holm\(1979\)A simple sequentially rejective multiple test procedure\.Scandinavian Journal of Statistics6\(2\),pp\. 65–70\.Cited by:[§V](https://arxiv.org/html/2608.10037#S5.p3.2)\.
- \[17\]S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, J\. Wang, C\. Zhang, S\. Yau, Z\. Lin, L\. Zhou,et al\.\(2024\)MetaGPT: meta programming for a multi\-agent collaborative framework\.InProceedings of the International Conference on Learning Representations,Vol\.2024,pp\. 23247–23275\.Cited by:[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[18\]X\. Hou, Y\. Zhao, Y\. Liu, Z\. Yang, K\. Wang, L\. Li, X\. Luo, D\. Lo, J\. Grundy, and H\. Wang\(2024\)Large language models for software engineering: a systematic literature review\.ACM Transactions on Software Engineering and Methodology33\(8\),pp\. 1–79\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1)\.
- \[19\]C\. Hsieh, S\. Chen, C\. Li, Y\. Fujii, A\. Ratner, C\. Lee, R\. Krishna, and T\. Pfister\(2023\)Tool documentation enables zero\-shot tool\-usage with large language models\.arXiv preprint arXiv:2308\.00675\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1)\.
- \[20\]H\. Huang, O\. Zheng, D\. Wang, J\. Yin, Z\. Wang, S\. Ding, H\. Yin, C\. Xu, R\. Yang, Q\. Zheng,et al\.\(2023\)ChatGPT for shaping the future of dentistry: the potential of multi\-modal large language model\.International Journal of Oral Science15\(1\),pp\. 29\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1)\.
- \[21\]K\. Huang, A\. Prabhakar, S\. Dhawan, Y\. Mao, H\. Wang, S\. Savarese, C\. Xiong, P\. Laban, and C\. Wu\(2025\)Crmarena: understanding the capacity of llm agents to perform professional crm tasks in realistic environments\.InProceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 3830–3850\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.13.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[22\]Y\. Huang, J\. Shi, Y\. Li, C\. Fan, S\. Wu, Q\. Zhang, Y\. Liu, P\. Zhou, Y\. Wan, N\. Gong,et al\.\(2024\)Metatool benchmark for large language models: deciding whether to use tools and which to use\.InProceedings of the International Conference on Learning Representations,pp\. 42978–43007\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.14.1),[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[23\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan\(2024\)Swe\-bench: can language models resolve real\-world github issues?\.InProceedings of the International Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.12.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[24\]J\. Y\. Khan, Md\. Tawkat Islam Khondaker, G\. Uddin, and A\. Iqbal\(2021\)Automatic detection of five api documentation smells: practitioners’ perspectives\.InProceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering,pp\. 318–329\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[25\]LangChain\(2024\)LangChain: build agents faster, your way\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://langchain.com/)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§II\-A](https://arxiv.org/html/2608.10037#S2.SS1.p2.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[26\]H\. Li, S\. You, F\. Di Palo, Y\. Qian, and A\. Jain\(2025\)Verification\-guided context optimization for tool calling via hierarchical llms\-as\-editors\.arXiv preprint arXiv:2512\.13860\.Cited by:[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[27\]M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. Li\(2023\)Api\-bank: a comprehensive benchmark for tool\-augmented llms\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 3102–3116\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.4.1),[§I](https://arxiv.org/html/2608.10037#S1.p1.1),[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§I](https://arxiv.org/html/2608.10037#S1.p9.1.3),[§IV\-A](https://arxiv.org/html/2608.10037#S4.SS1.p1.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[28\]F\. Lin, D\. J\. Kim, and T\. Chen\(2025\)Soen\-101: code generation by emulating software process models using large language model agents\.In2025 IEEE/ACM 47th International Conference on Software Engineering \(ICSE\),pp\. 1527–1539\.Cited by:[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[29\]M\. M\. Liu, D\. Garcia, F\. Parllaku, V\. Upadhyay, S\. F\. A\. Shah, and D\. Roth\(2025\)ToolScope: enhancing llm agent tool use through tool merging and context\-aware filtering\.arXiv preprint arXiv:2510\.20036\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p3.1)\.
- \[30\]W\. Maalej and M\. P\. Robillard\(2013\)Patterns of knowledge in api reference documentation\.IEEE Transactions on software Engineering39\(9\),pp\. 1264–1282\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[31\]P\. E\. McKnight and J\. Najab\(2010\)Mann\-whitney u test\.The Corsini encyclopedia of psychology,pp\. 1–1\.Cited by:[§V](https://arxiv.org/html/2608.10037#S5.p3.2)\.
- \[32\]N\. Nahar, S\. Zhou, G\. Lewis, and C\. Kästner\(2022\)Collaboration challenges in building ml\-enabled systems: communication, documentation, engineering, and process\.InProceedings of the 44th international conference on software engineering,pp\. 413–425\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[33\]K\. Nybom, A\. Ashraf, and I\. Porres\(2018\)A systematic mapping study on api documentation generation approaches\.InProceedings of the 44th Euromicro Conference on Software Engineering and Advanced Applications,pp\. 462–469\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[34\]OpenAI\(2023\)Introducing chatgpt\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://openai.com/blog/chatgpt)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1)\.
- \[35\]OpenAI\(2024\)GPT\-4o\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://openai.com/index/hello-gpt-4o/)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p5.1),[§II\-A](https://arxiv.org/html/2608.10037#S2.SS1.p2.1)\.
- \[36\]OpenAI\(2025\)Introducing codex\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://openai.com/index/introducing-codex/)Cited by:[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[37\]OpenAPIHub\(2024\)OpenAPIHub\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://www.openapihub.com/)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1)\.
- \[38\]S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez\(2024\)Gorilla: large language model connected with massive apis\.Advances in Neural Information Processing Systems37,pp\. 126544–126565\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.6.1),[§I](https://arxiv.org/html/2608.10037#S1.p1.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[39\]M\. Piccioni, C\. A\. Furia, and B\. Meyer\(2013\)An empirical study of api usability\.InProceedings of the ACM / IEEE International Symposium on Empirical Software Engineering and Measurement,pp\. 5–14\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[40\]Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.\(2024\)ToolLLM: facilitating large language models to master 16000\+ real\-world apis\.InProceedings of the International Conference on Learning Representations,Vol\.2024,pp\. 9695–9717\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.5.1),[§I](https://arxiv.org/html/2608.10037#S1.p1.1),[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[41\]C\. Qu, S\. Dai, X\. Wei, H\. Cai, S\. Wang, D\. Yin, J\. Xu, and J\. Wen\(2024\)Towards completeness\-oriented tool retrieval for large language models\.InProceedings of the 33rd ACM International Conference on Information and Knowledge Management,pp\. 1930–1940\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.15.1),[§I](https://arxiv.org/html/2608.10037#S1.p3.1)\.
- \[42\]C\. Qu, S\. Dai, X\. Wei, H\. Cai, S\. Wang, D\. Yin, J\. Xu, and J\. Wen\(2025\)From exploration to mastery: enabling llms to master tools via self\-driven interactions\.InProceedings of the International Conference on Learning Representations,pp\. 55675–55697\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§I](https://arxiv.org/html/2608.10037#S1.p9.1),[§IV\-A](https://arxiv.org/html/2608.10037#S4.SS1.p3.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[43\]RapidAPI\(2024\)RapidAPI\(Website\)Note:Accessed on May 13, 2026External Links:[Link](https://rapidapi.com/)Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1)\.
- \[44\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom\(2023\)ToolFormer: language models can teach themselves to use tools\.Advances in Neural Information Processing Systems36,pp\. 68539–68551\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[45\]H\. Shen, Y\. Li, D\. Meng, D\. Cai, S\. Qi, L\. Zhang, M\. Xu, and Y\. Ma\(2025\)Shortcutsbench: a large\-scale real\-world benchmark for api\-based agents\.InProceedings of the International Conference on Learning Representations,pp\. 77772–77797\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.10.1),[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[46\]L\. Shi, H\. Zhong, T\. Xie, and M\. Li\(2011\)An empirical study on evolution of api documentation\.InProceedings of the International Conference on Fundamental Approaches To Software Engineering,pp\. 416–431\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[47\]Z\. Shi, S\. Gao, L\. Yan, Y\. Feng, X\. Chen, Z\. Chen, D\. Yin, S\. Verberne, and Z\. Ren\(2025\)Tool learning in the wild: empowering language models as automatic tool agents\.InProceedings of the ACM on Web Conference,pp\. 2222–2237\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p3.1)\.
- \[48\]O\. Styles, S\. Miller, P\. Cerda\-Mardini, T\. Guha, V\. Sanchez, and B\. Vidgen\(2024\)Workbench: a benchmark dataset for agents in a realistic workplace setting\.arXiv preprint arXiv:2405\.00823\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.3.1),[§I](https://arxiv.org/html/2608.10037#S1.p5.1),[§I](https://arxiv.org/html/2608.10037#S1.p9.1.3),[§II\-A](https://arxiv.org/html/2608.10037#S2.SS1.p5.1.1),[§IV\-A](https://arxiv.org/html/2608.10037#S4.SS1.p1.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[49\]S\. Subramanian, L\. Inozemtseva, and R\. Holmes\(2014\)Live api documentation\.InProceedings of the 36th international conference on software engineering,pp\. 643–652\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[50\]Q\. Tang, Z\. Deng, H\. Lin, X\. Han, Q\. Liang, B\. Cao, and L\. Sun\(2023\)ToolAlpaca: generalized tool learning for language models with 3000 simulated cases\.ArXivabs/2306\.05301\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.7.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[51\]Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.\(2024\)Autogen: enabling next\-gen llm applications via multi\-agent conversations\.InProceedings of the 1st Conference on Language Modeling,Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p5.1),[§II\-A](https://arxiv.org/html/2608.10037#S2.SS1.p2.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[52\]C\. S\. Xia, Y\. Deng, S\. Dunn, and L\. Zhang\(2025\)Demystifying llm\-based software engineering agents\.Proceedings of the ACM on Software Engineering2\(FSE\),pp\. 801–824\.Cited by:[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[53\]Q\. Xu, F\. Hong, B\. Li, C\. Hu, Z\. Chen, and J\. Zhang\(2023\)On the tool manipulation capability of open\-source large language models\.ArXivabs/2305\.16504\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.9.1),[§I](https://arxiv.org/html/2608.10037#S1.p1.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[54\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press\(2024\)Swe\-agent: agent\-computer interfaces enable automated software engineering\.Advances in Neural Information Processing Systems37,pp\. 50528–50652\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.12.1),[§I](https://arxiv.org/html/2608.10037#S1.p1.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[55\]S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan\(2024\)τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.1.1),[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[56\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2022\)React: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p1.1),[§I](https://arxiv.org/html/2608.10037#S1.p5.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[57\]P\. Yu, W\. Liu, Y\. Yang, J\. Li, Z\. Zhang, X\. Feng, and F\. Zhang\(2026\)Benchmarking llm tool\-use in the wild\.arXiv preprint arXiv:2604\.06185\.Cited by:[TABLE I](https://arxiv.org/html/2608.10037#S1.T1.1.1.1.1.11.1),[§VI\-A](https://arxiv.org/html/2608.10037#S6.SS1.p1.1)\.
- \[58\]S\. Yuan, K\. Song, J\. Chen, X\. Tan, Y\. Shen, K\. Ren, D\. Li, and D\. Yang\(2025\)EasyTool: enhancing llm\-based agents with concise tool instruction\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 951–972\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p2.1),[§I](https://arxiv.org/html/2608.10037#S1.p3.1),[§I](https://arxiv.org/html/2608.10037#S1.p9.1),[§III\-C](https://arxiv.org/html/2608.10037#S3.SS3.p1.5),[§IV\-A](https://arxiv.org/html/2608.10037#S4.SS1.p3.1),[§VI\-B](https://arxiv.org/html/2608.10037#S6.SS2.p1.1)\.
- \[59\]A\. Zeng, X\. Lv, Z\. Hou, Z\. Du, Q\. Zheng, B\. Chen, D\. Yin, C\. Ge, C\. Huang, C\. Xie,et al\.\(2026\)Glm\-5: from vibe coding to agentic engineering\.arXiv preprint arXiv:2602\.15763\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p5.1),[§II\-A](https://arxiv.org/html/2608.10037#S2.SS1.p2.1)\.
- \[60\]L\. Zhuang, Q\. Zhang, H\. Zhou, Y\. Zhang, and X\. Huang\(2026\)Losemb: logic\-guided semantic bridging for inductive tool retrieval\.InProceedings of the ACM Web Conference,pp\. 3835–3846\.Cited by:[§I](https://arxiv.org/html/2608.10037#S1.p3.1)\.