Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse

# Silent Failures in Agent–Tool Interaction: An Audit of ToolUniverse
Source: [https://arxiv.org/html/2609.26836](https://arxiv.org/html/2609.26836)
###### Abstract

Agentic AI systems are increasingly adopting automated pipelines that integrate multiple tools\. While prior research and benchmarks have studied about task success and task completion of these agentic systems, the research about agent to tool interaction, specifically in biology agentic workflow is limited\. This study investigates specific failures in agent to tool interaction where a tool invocation appears successful, some or all of the information or functionality from the tool via API/ wrapper is incomplete or missing and there are no communications / notifications to the user or the agent about such missing information\. We call this a silent failures as the user or the agents are not aware that such failure has occurred\. For the purposes of this study we developed an audit mechanism to identify such silent failures in Agent to tool interaction, by examining 15 scientific tools \(and their associated API documentation and tool documentations\) integrated within ToolUniverse environment \(ToolUniverse serves as our experimental environment rather than the object of the study itself\)\. We structure our study around 7 failure locus characterising where the failure occurs in the chain\. We observed 91 failures \(manually validated post LLM based candidate discovery and automated testing\), most frequent of them being missing data or fields and inconsistencies in search, filtering or ranking criteria\. Most of the 91 failures occurred in API layer \(51\) or wrapper layer \(25\), with a potential of silent failure amplification downstream\. The results show that silent failures originate upstream of the event and propagate downstream into apparently valid scientific outputs\. We propose a concept of contextual reliability to handle such failures and suggest mechanisms for testing, disclosing, monitoring, and measuring such failures across the agent–tool interaction pipeline\.

## 1Introduction

#### Agent\-Tool interaction:

AI systems for science are increasingly connecting large language models \(LLMs\) and agents to domain\-specific databases, wrappers or APIs, computational or other specialized tools, and ever growing literature, to provide stronger research and solutions for domain\-specialists\. Such systems in biology including Virtual Lab, ToolUniverse, and Biomni demonstrate the use of tools within agentic scientific workflows\[[35](https://arxiv.org/html/2609.26836#bib.bib1),[10](https://arxiv.org/html/2609.26836#bib.bib2),[14](https://arxiv.org/html/2609.26836#bib.bib3),[13](https://arxiv.org/html/2609.26836#bib.bib4)\]\. In such systems, the agents integrated with large multimodal models, not only handle multi\-modality of data, but also search, select, collaborate, collate, interpret, and combines external scientific information and tools\. However, the reliability of the interaction between an agent and these tools remains insufficiently evaluated\. Existing evaluations of agentic systems commonly emphasize task outcome success including resolving an issue in SWE\-bench, completing an interactive task in AgentBench, or executing a valid tool\-use trajectory in ToolBench\[[17](https://arxiv.org/html/2609.26836#bib.bib9),[23](https://arxiv.org/html/2609.26836#bib.bib10),[30](https://arxiv.org/html/2609.26836#bib.bib11)\]\. These evaluations do not necessarily establish whether the information returned by the tool is complete, appropriately qualified, correctly represented, or correctly interpreted by the agent\. A tool may return a valid response while the final answer is incomplete, incorrectly qualified, or scientifically misinterpreted\. This problem is particularly consequential in biology, where provenance, evidence strength, experimental status, organism and disease context, and completeness can determine the meaning of a result\. At scale, interactions that cannot be individually inspected accumulate verification debt\[[26](https://arxiv.org/html/2609.26836#bib.bib5),[36](https://arxiv.org/html/2609.26836#bib.bib6),[38](https://arxiv.org/html/2609.26836#bib.bib7)\]\.We choose, information flow from the agent to tool interaction pipeline\. In specific, we were interested in understanding where the pipeline fails and the message stays silent for the agent or for the user\. Prior work on LLM\-agent debugging shows that errors can cascade across planning, memory, action, and system\-level components\[[40](https://arxiv.org/html/2609.26836#bib.bib8)\]; provenance related work also highlight tracing data and transformations of such data across the scientific workflows\[[2](https://arxiv.org/html/2609.26836#bib.bib12),[32](https://arxiv.org/html/2609.26836#bib.bib13),[7](https://arxiv.org/html/2609.26836#bib.bib14)\]\. We refer to the subset of such errors that remain undisclosed as*silent failures*\.

#### Defining silent failure in agent\-tool interaction:

We define a silent failure as an interaction in which \(1\) the tool invocation appears successful, \(2\) the returned information is incomplete, transformed, ambiguous, or otherwise insufficient for the intended task, and \(3\) the limitation is not adequately disclosed to the downstream user or agent\. Unlike a loud failure, where an API timeout reported to the user, a silent failure can arise when an API returns its first 1,000 records \(out of 10000 records\) and the agent reports “the records\.” Examples include a truncated result presented as complete results, an unavailable filter information silently generalized by an agent, a missing qualifier treated as absence of information, and a score or evidence hierarchy interpreted incorrectly\. Silent failures can therefore propagate across system layers\. An upstream limitation that is relatively benign at the tool or API layer can become a more consequential error when the agent interprets the partial output as complete evidence\. We refer to this as silent amplification\.

#### Research theme:

Our focus is distinct from general studies of an agent’s internal reasoning failures or its final task success\. We audit the*agent–tool interaction*: whether the agent, wrapper, API, and underlying resource preserve the information, qualifiers, provenance, scope, and semantics needed for the agent’s claim\. Thus, even an agent that plans correctly and completes a task can silently amplify a limitation introduced at an external tool boundary\. These risks are especially consequential in biology, where downstream reuse and interpretation depend on rich, contextual metadata and provenance\[[39](https://arxiv.org/html/2609.26836#bib.bib15)\]\.

User intent→\\rightarrowAgent→\\rightarrowWrapper→\\rightarrowAPI→\\rightarrowTool/Data→\\rightarrowAPI→\\rightarrowWrapper→\\rightarrowAgent→\\rightarrowClaim

Figure 1:The agent–tool interaction chain\. A response can be technically successful while information, qualifiers, provenance, scope, or meaning is lost at one or more boundaries\.Our central question is:*when an agent successfully uses a scientific tool, how reliably does the interaction preserve the information, qualifiers, context, and meaning required to support the intended conclusion?*\.

Our research questions are:

- •RQ1:How prevalent are potential silent failures when scientific tools are used through an agentic interface?
- •RQ2:At which locus in the Agent–Wrapper–API–Tool chain do these failures occur?
- •RQ3:Which information, qualifier, context, and semantic properties are most vulnerable?
- •RQ4:How often does the agent preserve, disclose, or amplify upstream limitations?

We address these research questions by examining tools integrated within ToolUniverse environment\.

#### Failure loci in agent\-tool interactions:

We organize our research in Failure Locus \(the point at which the failure originates thematically\)\. Our failure locus taxonomy consists of the following: \(L1\)*Tool Limitations:*inherent limitation of the tool or data or meaning associated with the tool; \(L2\)*API Gap:*tool/webUI has the capability/ the feature, however, it is completely missing from the API \(no endpoint / no way to call it\); \(L3\)*API Partial Gap:*tool/webUI has the capability/ the feature, however, it is partially missing from the API\. For instance, the endpoint exists, but certain fields are not covered as part of the API in comparison to the tool; \(L4\)*Wrapper Gap:*tool/webUI and the API has the capability/ the feature, however, no wrapper exposes such capability; \(L5\)*Wrapper Partial Gap:*tool/webUI and the API has the capability/ the feature, however, wrapper partially omits parameters, fields, or formats which the API returns; \(L6\)*Agent Usability Gap:*Wrapper provided an usable output, but the agent could not use it; and \(L7\)*Agent Interpretation Gap:*The AI agent received the wrapper output but misinterpreted it\. This includes overstating, understating, treating incomplete as complete or vice versa\.

## 2Experimental environment and methodology

![Refer to caption](https://arxiv.org/html/2609.26836v1/prism-uploads/Flowchart.png)

Figure 2:Flowchart of experimental methodology\.#### ToolUniverse setup and selection of tools:

ToolUniverse was set up using the official MCP guidance with the AntiGravity CLI agent as the experimental environment\. We randomly sampled 63 main tools having over 285 associated tools in ToolUniverse\. In this study,*main tool*refers to the underlying biological database or computational resource and the individual ToolUniverse wrapper or function is called as*tool*or*wrapper*\. From the initial sample, 15 tools \(having 104 associated ToolUniverse tools\) were selected for detailed analysis based on the identification of documented candidate failures that could be subjected to detailed investigation\.

#### Failure candidate discovery:

For each sampled tool, we assembled a tool context containing available documentation and implementation information\. The context was provided to the Cursor agent as a combination of URLs, extracted content, and structured JSON\. Sources included tool documentation; web UI details; API documentation; user manual: manual, tool guide, beginner’s manual, etc; release notes for both the API and web UI \(if available\); reported GitHub issues\.

Based on this context, the Cursor agent generated potential failures by identifying conflicts, contract disagreements, implementation issues, and Web/API/wrapper inconsistencies\. The systematic comparison considered functionality, fields, parameters, filtering, ranking, pagination, formats, semantics, provenance, and temporal changes\. To address our central question we examined properties like \- requested fields/results are present/ available; conditions attached to the information are preserved; source/evidence attribution is retained; boundaries of the retrieved result are clear; and the semantic interpretation of the returned information/ outcome is preserved\. Preserving meaning is as important as preserving the information\. For instance, collapsing phospho\-SMAD3 to SMAD3, flattening CIViC evidence and assertions, or exposing a low\-level genome\-quality flag without its specific interpretation can impact the conclusions arrived by the agent\.

We also examinedevolutionary context, including changes in APIs, Web UIs, documentation, and tool releases that could alter data scope, assumptions, interpretation, or meaning\. This was used to identify candidate failures involving temporal changes or changes in the context in which returned information should be interpreted\.

#### Test\-case and query generation:

For each candidate failure, concrete test cases and queries were generated to determine whether the observed limitation could be reproduced through ToolUniverse\. Test\-case and query generation was iteratively refined through manual verification before automating\.

- •Tool\-level specification:Queries identified the underlying main tool rather than prescribing a specific ToolUniverse tool, allowing the agent to select the relevant functionality\.
- •Exact tool names:Tool names were restricted to names present in ToolUniverse\.
- •Complete inputs:Each test case contained the specific identifiers, parameters, or other inputs required to execute the test\. For example, a test case specified an accession such asDP00086rather than an underspecified entity such as TP53\.
- •Concrete scientific inputs:Test cases used concrete biological entities, identifiers, datasets, or other domain\-specific inputs rather than hypothetical examples\.
- •Failure isolation:Queries were designed to test the target failure rather than ask the agent to independently determine whether a failure existed\. In particular, queries did not instruct the agent to compare its output against the raw API, assess completeness, or simulate infrastructure failures to separate confounding between the target tool failure and failure introduced by the test query

For example, a test query could request: "Use the ToolUniverse toolDisProt\_get\_entrywithaccession=’DP00086’\. Report the full tool output, listing every field returned\. Do not invent fields\."; rather than asking the agent to determine whether the output was complete or to compare it against an external source\. We choose this distinction because the objective of the test was to observe the behavior of the Agent–Tool interaction, rather than to delegate failure diagnosis to the agent itself\.

#### Repeated experimental runs:

Each candidate failure was evaluated through repeated agent executions\. An initial test query was generated for each candidate\. If the initial execution was classified as a failure, four additional queries targeting the same failure were executed\. If the initial execution was classified as pass or unverified, two additional query was executed\. Additional queries used different concrete scientific inputs where possible to assess whether the observed behavior generalized beyond a single test case\.

#### Failure validation and locus attribution:

The Cursor agent was used as an automated evaluator of the execution traces\. For each run, it classified the result as either*Fail*\(the candidate failure was observed\) or*False Positive*\(the candidate failure was not observed\)\. When the evaluator could not establish either outcome from the available evidence, the case was marked*Unverified / Needs Additional Validation*\. Each validated failure was subsequently mapped to the failure\-locus taxonomy defined earlier in section 1, distinguishing tool limitations, API gaps, wrapper gaps, agent usability failures, and agent interpretation failures\.

#### Manual validation:

While the whole process for followed for 63*main tools*, two authors manually reviewed 15 main tools \(198 test cases\) to validate the LLM\-driven/ cursor driven verification\. The author tested reviewed the outcomes from cursor evaluation and reperformed them using API and ToolUniverse\-Antigravity\-CLI agent calls to validate the automated evaluations\. The results represented in the report covers only the human verified outcomes from the 15 main tools\. Manual validation of the failure test cases involved checking the failure test cases for the following: \(1\)*Fact Checking*: Is the selected candidate appropriate for the given failure scenario? This was assessed by manually validating the candidate through analyzing the availability of results for the candidate in the Web UI\. \(2\)*Judgement Validation*: Is the LLM judgement valid for the given failure scenario, and are there other possibilities that the LLM might not have understood in terms of the actual tool functionalities and features, which might contribute to the current judgement rather than an actual failure?

#### Failure measurement and rubrics:

Each failure was classified along the following ’analysis dimensions’: completeness, correctness, appropriateness, consistency, reliability, attribution, interpretation, relevance, context, and purpose alignment\. Appendix[A](https://arxiv.org/html/2609.26836#A1)describes the dimensions and rubrics in detail\.

## 3Results

Failures were distributed across multiple layers of the Agent–Wrapper–API–Tool interaction\. The following analysis summarizes the distribution of confirmed failure cases across the seven failure loci, normalized within each tool\. The loci correspond to the failure taxonomy defined in Section 1\.

![Refer to caption](https://arxiv.org/html/2609.26836v1/prism-uploads/Updated_Fig_3.png)

Figure 3:\(A\) Distribution of grades across each manually\-validated tool\. \(B\) Distribution of failures across each failure locus \(failure locus of 3 cases could not be determined\.\)As shown in Figure[3](https://arxiv.org/html/2609.26836#S3.F3), 7 out of 15 tools exhibit confirmed failures on more than 50% of test cases \- ExpressionAtlas \(75%\), BV\-BRC \(67%\), AMPSphere \(67%\), CTIS \(53%\), ChIP\-Atlas \(53%\), KEGG \(53%\) and Reactome \(53%\)\. The dominant failure loci for most tools was API partial gap \(L3\) and wrapper partial gap \(L5\)\. The dominant failure loci for most tools were API partial gap \(L3\) and wrapper partial gap \(L5\)\. These two loci accounted for the largest*share of human\-verified failures*in BiGG Models \(100%\), BV\-BRC \(100%\), CIViC \(100%\) and Reactome \(88%\)\. These results indicate that many failures did not arise because an API was entirely unavailable, but because the API provided only a partial representation of the underlying resource, including limitations in fields, filtering, pagination, coverage, or other capabilities\. This indicates that partial functionality can produce a valid response while restricting the information available to the agent\. The interaction can therefore appear successful even though the returned information is insufficient for the intended scientific conclusion\. When the agent treats this response as sufficient evidence, the upstream limitation can be converted into a downstream scientific claim\. We refer to this propagation as silent amplification\. Other tools also had confirmed failure locus outside L3 and L5\. Tool limitations \(L1\) accounted for 25% of confirmed cases in ChIP\-Atlas, 20% in AMPSphere\. API gap \(L2\) accounted for 25% in both CTIS and KEGG\. Wrapper complete gap \(L4\) appeared in KEGG \(13%\) and AMPSphere \(10%\)\. Agent interpretation gap \(L7\) appeared in CryoET Data Portal \(33%\), ClinicalTrials\.gov \(25%\), ChIP\-Atlas \(13%\)\.

Failures at the agent\-level loci were comparatively uncommon in this analysis\. Agent Usability Gap accounted for no confirmed failures in any of the 15 tools, while Agent Interpretation Gap \(L7\) appeared only in AMPSphere \(10%\), ChIP\-Atlas \(13%\), ClinicalTrials\.gov \(25%\), CryoET Data Portal \(33%\), CTIS \(13%\), and KEGG \(13%\)\. This distribution should not be interpreted as evidence that agent\-level failures are absent\. Rather, within the human vertified cases identified in this study, most failures could be traced to limitations or gaps upstream of agent interpretation\. Also the failures are deeply tied to the test cases generated\. The volume of test cases generated based on the failure candidate discovery and hence, the proportion of failures and absence thereof need to be considered in context of such failure candidate discovery process\. We emphasise that the results are not representative of exhaustive failures in agent\-tool interaction within the ToolUniverse environment for the identified 15 tools, but a illustrative representation of silent failures that exists in such environment\.

![Refer to caption](https://arxiv.org/html/2609.26836v1/prism-uploads/Updated_Fig_4.png)Figure 4:\(A\) Issue occurrences across the analysis dimensions\. Individual test cases may contribute to more than one dimension\. \(B\) Observed types of issues\. Individual test cases may exhibit to more than one issue\.Figure[4](https://arxiv.org/html/2609.26836#S3.F4)shows the distribution of issue occurrences across the analysis dimensions\. The 91 evaluated cases generated 134 issue occurrences because individual test cases could involve multiple dimensions\. Completeness was the most frequently affected dimension, with 28 occurrences, followed by Ranking and Relevance \(21\) and Result Appropriateness \(20\)\. Source and Data Attribution \(18\), Interpretation Variabilities \(13\), Reliability of Information \(13\), and Consistency \(11\) were also frequently represented\. The concentration of issues in completeness indicates that failures frequently involve limitations in the amount, scope, or coverage of information available to the agent rather than complete failure to retrieve a result\. Interpretation and result appropriateness were the next most frequent dimensions, indicating that obtaining information does not necessarily ensure that it can be appropriately interpreted or used for the intended scientific task\.

In terms of types of issues \(Fig 4B\) that were most frequent issue types with 38 occurances,indicating that failures often involved incomplete information rather than complete retrieval failure\. Also, search, filtering, and ranking criteria were the second most frequent occurances \(20 instances\) exhibiting the issues contributed by qualifiers and limitations in retrieval of the complete information\. Missing complete functionality \(8\) and data mismatch/correctness/ambiguity \(6\) were also observed, showing that available interfaces did not always provide the functionality or representations required by the test case\. Agent misinterpretation \(5\) and error handling/ false success declarations \(4\) were less frequent but were relevant as they modulate silent failures into confident result outcomes\.

The most consequential cases are propagating phenomena: a limitation at one boundary changes the claim at another\. EMDB illustrates API capping becoming a completeness claim; ClinicalTrials\.gov illustrates missing filtering becoming inappropriate inclusion; ChIP\-Atlas illustrates score or biological\-state representation becoming semantic misinterpretation; and CIViC illustrates evidence\-hierarchy flattening becoming epistemic confusion\. In each case, the agent can amplify rather than merely inherit the limitation\.

## 4Representative examples

#### L1: Tool Limitations

Historically, the Expression Atlas tool functioned by re\-hosting GTEx data from other sources under their study\. They used ArrayExpress\-style accessions \(E\-MTAB\-xxxx\) but when they started hosting their own GTEx studies, they changed to a new canonical identifier \(E\-GTEx\-xxxx\)\. Some samples were rebranded with the new format while their legacy representations still persist in URLs\. However, for these rebranded samples, there is no alias table that resolves the older pattern of identifier to the new ones\. And if searched through legacy IDs, the API returns 0 results unlike the new IDs \(for the same GTEx study as the legacy ID\), the API returned appropriate results\. In another instance, the search features of the BiGG models, for a search query “Glucose” using the genes type, returns some non\-glucose metabolizing genes in the top hits\. This is attributable to the non\-specific substring search logic in the BiGG model application\. An agent might misidentify irrelevant genes based on these substring matches, leading to mistaken interpretations\.

#### L2: API Gap

The ClinGen Web UI clearly supports searching by drug name or RXNORM ID, but the underlying API endpoints \(/api/validity and /api/dosage\) lack the capability to handle such a query parameter, as evidenced by the API returning the full unfiltered dataset instead of filtering by the requested drug name\. This could lead the agent to misinterpret that the whole dataset has relevant experiments for the requested drug name\. The ChIP\-Atlas Tool Web UI provides access to ’Annotation tracks’ \(308 tracks\) and the new ’Experiment Comparative Profile’ feature, but neither functionality is exposed via any programmatic API endpoints\. The agent does not have access to statistical summaries through these features, which might force it search manually through other means or make an assumption from incomplete data\.

#### L3: API partial gap

The official BiGG Models API endpoint for individual model details returns a null value for the ’organism’ field for the model ’iRC1080’, despite the organism information being available and displayed in other parts of the BiGG web interface \(such as search results\)\. This indicates the API provides incomplete data contract coverage for model details\. The AMPSphere tool contains information on the existence of antimicrobial and hemolytic activity probabilities for antimicrobial peptides in the Web UI\. However, the official tool API omits these fields entirely from the response, meaning that an agent accessing this tool for an AMP would have no idea of the predicted bioactivity of these peptides\.

#### L4: Wrapper Gap

The official clinicaltrials\.gov API supports complex age range filtering to filter for a particular age group using the filter\.advanced parameter\. However, the ToolUniverse wrapper ‘ClinicalTrials\_search\_studies” fails to adopt these filtering strategies, forcing the agent to rely on a broader free\-text search string, which might lead to less precise results\. The ChIP Atlas tool offers a platform to perform Enrichment analysis, and has recently begun to accept gene count tables as an input as well in the tool API\. The gene count tables provide an added advantage because it does not require the user to provide a gene list from thresholding a differential expression data\. Instead, the raw experimental values can directly be provided to the platform and it computes the enrichment statistics from a continuous expression rather than a binary gene list\. However, the ToolUniverse wrapper ’ChIPAtlas\_enrichment\_analysis’ lacks any parameter to accept or pass these file\-based inputs, which implies that the agent has to rely on the original gene list for enrichment analysis\.

#### L5: Wrapper partial gap

The CryoET data portal organizes data such that a single submitted project can have multiple child datasets associated to it\. The GraphQL API supports the retrieval of these child datasets for a particular deposition\. But the ToolUniverse wrapper ‘CryoET\_list\_depositions’ completely omits this field from its return schema\. The absence of child datasets would prevent an agent from discovering the hierarchical relationship between multiple datasets and a project\. Another instance of missing partial information through the wrapper was observed in the ClinVar tools\. The API response provides the ’ContributesToAggregateClassification’ attribute required to identify which submissions contribute to the aggregate\. However, the ToolUniverse wrapper ’ClinVar\_get\_submitted\_records’ parses the XML and fails to map or expose this field in its output schema, preventing the AI agent from distinguishing contributing vs\. non\-contributing records\.

#### L7: Agent interpretation gaps

The ChIP Atlas tool has a software, MACS2, to process raw sequencing data to identify regions which are functionally enriched\. The statistical significance is computed using a logarithmic formula \(\-10 \* log10\(Q\-value\)\), as established in the tool’s documentation\. When the agent was provided with a valid BED data via the wrapper, it misinterpreted the score to be a generic confidence score, failing to apply the logarithmic definition\. KEGG’s module states that a space or a plus sign, representing a connection in the pathway or the molecular complex, is treated as an AND operator and a comma, used for alternatives, is treated as an OR operator\. And the module completeness requires that every top\-level parenthetical block \(AND\-connected step\) have at least one satisfied member \(OR\-connected alternative\) present\. The agent correctly identifies the boolean ’OR’ behavior of commas in the module definition but misinterprets the module completeness rule by stating that missing a single K\-number within a group does not compromise functionality, ignoring the fact that module completeness requires at least one present K\-number for every parenthetical group \(step\) in the module definition\. Representative instances linking reliability dimensions to failure loci are provided in Appendix[B](https://arxiv.org/html/2609.26836#A2)\(Table[2](https://arxiv.org/html/2609.26836#A2.T2)\)\.

## 5Discussion

#### Why silent failures occur?

This study was designed to identify and characterize silent failures in agent–tool interactions, rather than to establish their engineering or organizational root causes\. We therefore treat the following as plausible ecosystem\-level contributors, not as causal explanations for every observed case\. First, rapidly evolving agent architectures, models, APIs, and tool integrations can outpace systematic testing of cross\-component behaviors; recent work on multi\-agent systems similarly identifies specification and system\-design failures as important sources of failure\[[3](https://arxiv.org/html/2609.26836#bib.bib31)\]\. Second, information flow is distributed across tool providers, API maintainers, wrapper developers, and agent developers, creating coordination and dependency\-stability challenges\[[1](https://arxiv.org/html/2609.26836#bib.bib32),[22](https://arxiv.org/html/2609.26836#bib.bib33)\]\. Third, scientific resources and their APIs evolve through updates, revisions, and changing documentation; empirical studies of Web and microservice APIs show that versioning, change\-impact analysis, and communicating changes to consumers remain persistent challenges\[[33](https://arxiv.org/html/2609.26836#bib.bib34),[22](https://arxiv.org/html/2609.26836#bib.bib33)\]\. Finally, when an API or community\-maintained wrapper exposes only part of a tool’s Web functionality, the integration can remain technically functional while still being incomplete for an agent’s intended task\. These factors motivate evaluating reliability across the complete Agent–Wrapper–API–Tool chain rather than treating any single component as the sole source of failure\.

#### What does it mean?

Silent failure is a property of an interaction pathway: the relevant unit of analysis is not an API, wrapper, or agent in isolation, but the complete Agent–Wrapper–API–Tool chain\. Provenance research similarly treats the origin, transformation, and movement of information across a workflow as essential to interpreting its outputs\[[2](https://arxiv.org/html/2609.26836#bib.bib12),[32](https://arxiv.org/html/2609.26836#bib.bib13),[7](https://arxiv.org/html/2609.26836#bib.bib14),[18](https://arxiv.org/html/2609.26836#bib.bib35)\]\. A silent failure therefore cannot always be attributed to the underlying tool alone\. Nor does a successful tool call establish that the agent has completed the task correctly or used the returned result faithfully; diagnostic agent evaluations distinguish, for example, calling a needed tool from ignoring its result or fabricating an unsupported output\[[30](https://arxiv.org/html/2609.26836#bib.bib11),[23](https://arxiv.org/html/2609.26836#bib.bib10),[34](https://arxiv.org/html/2609.26836#bib.bib36)\]\.

Consequently, an audit must establish whether all information required for the task was retrieved, whether relevant qualifiers and metadata were retained, and whether the output supports the intended scientific purpose\. This is particularly important in biological workflows, where provenance, context, and metadata shape the appropriate reuse and interpretation of data\[[39](https://arxiv.org/html/2609.26836#bib.bib15),[18](https://arxiv.org/html/2609.26836#bib.bib35)\]\. Unlike explicit errors, partial results remain available for downstream reasoning\. Omitted qualifiers—such as evidence type, organism specificity, or search scope—can therefore alter an agent’s interpretation without preventing execution\. Truncation, unavailable filtering, missing metadata, and ambiguous semantics can each propagate through the interaction chain and be amplified into a stronger downstream claim than the retrieved evidence supports\.

#### Why does this matter?

Silent failures have four important implications\. First, scientific workflows often compose multiple tools, so a limitation at one stage can propagate into later retrieval, analysis, or interpretation unless its provenance is retained\[[32](https://arxiv.org/html/2609.26836#bib.bib13),[7](https://arxiv.org/html/2609.26836#bib.bib14),[6](https://arxiv.org/html/2609.26836#bib.bib39)\]\. Second, verifying an apparently successful agent output can impose a substantial burden on users, particularly when the system appears reliable enough to encourage over\-reliance on automated recommendations\[[29](https://arxiv.org/html/2609.26836#bib.bib37),[11](https://arxiv.org/html/2609.26836#bib.bib38)\]\. Third, an apparent efficiency benefit can be offset by downstream checking, reanalysis, or rework when a limitation is discovered only after its output has been reused; provenance and workflow\-reproducibility research emphasizes the importance of tracing such divergences to support diagnosis and reuse\[[25](https://arxiv.org/html/2609.26836#bib.bib40),[7](https://arxiv.org/html/2609.26836#bib.bib14)\]\. Fourth, a silent failure may yield a fluent, coherent answer without establishing factual correctness, faithfulness to retrieved information, or adequacy of evidence for the stated conclusion\[[16](https://arxiv.org/html/2609.26836#bib.bib41)\]\.

#### What can be done?

We believe that users cannot reasonably be expected to understand the implementation and limitations of every tool available in an expanding agentic ecosystem\. While we believe skepticism is needed for the users while they leverage agentic pipelines for scientific workflows, we foresee four clear directions as future work in this space\. \(1\) Develop measurable contextual reliability indicators rather than binary tool\-success measures\. \(2\) Develop automated tests for qualifier preservation; completeness; provenance; scope; semantic preservation; and limitation disclosure\.\(3\) Develop runtime triggers and disclosures that alert agents/users when an interaction crosses a known reliability boundary\. Tool limitations should become part of the agent’s evidence state\. \(4\) Develop mechanisms for propagating tool limitations through wrappers and agents so that they remain visible in the final scientific output\. \(5\) Enable testing and evaluation of tool configurations at the time of deployment and periodically thereafter\. The testing shall cover API availability, parameter mapping, filed preservation, filtering, pagination, output formats, error handling, semantic interpretation etc\. \(6\) Encourage tools and wrappers to explicitly disclose \(in machine\-readable formats\) the functionalities that are unavailable, differences between web and API interfaces, logics and ranking mechanisms adopted in the API, unsupported parameters etc\. \(7\) Establish tool ecosystem needs monitoring mechanism\. A bug, failure or even adversary in one of the tools can impact the other tools in a given task or a set of tasks\. Also, given the tools are managed decentrally having a collective view on periodic maintenance, bug resolutions and updates can become relevant indicators for reliability\.

*Our proposition of contextual reliability:*Contextual reliability is the extent to which an Agent–Tool interaction preserves and communicates the information, qualifiers, provenance, scope, and meaning required to support the intended scientific conclusion, across the interaction chain\. Unlike conventional tool reliability, which may focus on whether a tool call executes successfully or returns valid data, contextual reliability evaluates whether the returned information remains sufficient and correctly interpretable in context \(e\.g a response has low contextual reliability if relevant information is truncated, qualifiers are lost, provenance is unavailable, scope is unclear, or the agent assigns a meaning not supported by the retrieved evidence\)\.

## 6Limitations and conclusion

We recognize that our study is not exhaustive and has certain limitations\. This study evaluates a ToolUniverse\-specific environment, and deep analysis covers 15 tools rather than the entire tool ecosystem\. Our approach to candidate generation and primary adjudication use LLMs, while manual validation covers only a subset of cases\. While one of the authors have reviewed the outcomes of the audit results and reperformed tests, the candidate generation may have missed some aspects which leverged LLM for such task\. The APIs, wrappers, interfaces, and data evolve, so individual observations may change with tool versions, its not static and our tests may vary with the changes\. Establishing that a failure is silent is inherently harder than detecting an explicit error, we have approached it in a certain way and our approach is not exhaustive\. The results are an empirical characterization of observed risks, not an exhaustive estimate of all agent–tool failures and as stated earlier in our approach the candidate generation may influence the proportion of failures observed and hence proportion of failures are not to be considered as representative of typical silent failures in agent\-tool interaction\.

In this study, we empirically characterize silent failures in agent–tool interactions across 15 scientific tools and 198 documented candidate cases\. We introduce a failure\-locus taxonomy spanning tool limitations, API gaps, wrapper gaps, agent usability, and agent interpretation\. We identified recurring mechanisms through which partial or transformed tool outputs become downstream scientific claims, including quiet subset presentation and silent amplification and we propose contextual reliability as a mechanism for evaluating whether information, qualifiers, provenance, scope, and meaning remain sufficient for the intended scientific conclusion\. We contribute these via a focused study of tools integrated within ToolUniverse\. We would like to clarify that ToolUniverse serves as our experimental environment rather than the object of the study itself\. We use its collection of scientific tools to systematically examine failure modes that can arise in agent–tool interactions\.

## References

- \[1\]C\. Bogart, C\. Kästner, and J\. Herbsleb\(2015\)When it breaks, it breaks: how ecosystem developers reason about the stability of dependencies\.InProceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering Workshop,pp\. 86–89\.External Links:[Document](https://dx.doi.org/10.1109/ASEW.2015.21)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px1.p1.1)\.
- \[2\]P\. Buneman, S\. Khanna, and W\. Tan\(2001\)Why and where: a characterization of data provenance\.InDatabase Theory — ICDT 2001,Lecture Notes in Computer Science, Vol\.1973,pp\. 316–330\.External Links:[Document](https://dx.doi.org/10.1007/3-540-44503-X%5F20)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px2.p1.1)\.
- \[3\]M\. Cemri, M\. Z\. Pan, S\. Yang, L\. A\. Agrawal, B\. Chopra, R\. Tiwari, K\. Keutzer, A\. Parameswaran, D\. Klein, K\. Ramchandran, M\. Zaharia, J\. E\. Gonzalez, and I\. Stoica\(2025\)Why do multi\-agent LLM systems fail?\.External Links:2503\.13657,[Document](https://dx.doi.org/10.48550/arXiv.2503.13657)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px1.p1.1)\.
- \[4\]ClinGen Consortium\(2025\)The clinical genome resource \(ClinGen\): advancing genomic knowledge through global curation\.Genetics in Medicine27\(1\),pp\. 101228\.External Links:[Document](https://dx.doi.org/10.1016/j.gim.2024.101228)Cited by:[§C\.3](https://arxiv.org/html/2609.26836#A3.SS3.p1.1)\.
- \[5\]D\. Croft, G\. O’Kelly, G\. Wu, R\. Haw, M\. Gillespie, L\. Matthews, M\. Caudy, P\. Garapati, G\. Gopinath, B\. Jassal, S\. Jupe, I\. Kalatskaya, S\. Mahajan, B\. May, N\. Ndegwa, E\. Schmidt, V\. Shamovsky, C\. Yung, E\. Birney, H\. Hermjakob, P\. D’Eustachio, and L\. Stein\(2011\)Reactome: a database of reactions, pathways and biological processes\.Nucleic Acids Research39\(Suppl\. 1\),pp\. D691–D697\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkq1018)Cited by:[§C\.15](https://arxiv.org/html/2609.26836#A3.SS15.p1.1)\.
- \[6\]V\. Cuevas\-Vicenttín, S\. Dey, S\. Köhler, S\. Riddle, and B\. Ludäscher\(2013\)Scientific workflows and provenance: introduction and research opportunities\.External Links:1311\.4610,[Document](https://dx.doi.org/10.48550/arXiv.1311.4610)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px3.p1.1)\.
- \[7\]S\. B\. Davidson and J\. Freire\(2008\)Provenance and scientific workflows: challenges and opportunities\.InProceedings of the 2008 ACM SIGMOD International Conference on Management of Data,pp\. 1345–1350\.External Links:[Document](https://dx.doi.org/10.1145/1376616.1376772)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px3.p1.1)\.
- \[8\]U\. Ermel, A\. Cheng, J\. X\. Ni, J\. Gadling, M\. Venkatakrishnan, K\. Evans, J\. Asuncion, A\. Sweet, J\. Pourroy, Z\. S\. Wang, K\. Khandwala, B\. Nelson, D\. McCarthy, E\. M\. Wang, R\. Agarwal, and B\. Carragher\(2024\)A data portal for providing standardized annotations for cryo\-electron tomography\.Nature Methods21\(12\),pp\. 2200–2202\.External Links:[Document](https://dx.doi.org/10.1038/s41592-024-02477-2)Cited by:[§C\.6](https://arxiv.org/html/2609.26836#A3.SS6.p1.1)\.
- \[9\]European Medicines Agency\(n\.d\.\)Clinical trials in the european union\.Note:Clinical Trials Information System \(CTIS\) public portalAccessed 2026\-09\-07External Links:[Link](https://euclinicaltrials.eu/)Cited by:[§C\.10](https://arxiv.org/html/2609.26836#A3.SS10.p1.1)\.
- \[10\]S\. Gao, R\. Zhu, P\. Sui, Z\. Kong, S\. Aldogom, Y\. Huang, A\. Noori, R\. Shamji, K\. Parvataneni, T\. Tsiligkaridis, and M\. Zitnik\(2025\)ToolUniverse: an open platform for democratizing AI scientists\.External Links:2509\.23426,[Document](https://dx.doi.org/10.48550/arXiv.2509.23426)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1)\.
- \[11\]K\. Goddard, A\. Roudsari, and J\. C\. Wyatt\(2012\)Automation bias: a systematic review of frequency, effect mediators, and mitigators\.Journal of the American Medical Informatics Association19\(1\),pp\. 121–127\.External Links:[Document](https://dx.doi.org/10.1136/amiajnl-2011-000089)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px3.p1.1)\.
- \[12\]M\. Griffith, N\. C\. Spies, K\. Krysiak, J\. F\. McMichael, A\. C\. Coffman, A\. M\. Danos, B\. J\. Ainscough, C\. A\. Ramirez, D\. T\. Rieke, L\. Kujan,et al\.\(2017\)CIViC is a community knowledgebase for expert crowdsourcing the clinical interpretation of variants in cancer\.Nature Genetics49\(2\),pp\. 170–174\.External Links:[Document](https://dx.doi.org/10.1038/ng.3774)Cited by:[§C\.4](https://arxiv.org/html/2609.26836#A3.SS4.p1.1)\.
- \[13\]GXL\(2026\)Making biological databases agent\-native with Paperclip\.Note:GXL BlogAccessed 2026\-08\-15External Links:[Link](https://gxl.ai/blog/adding-biology-databases)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1)\.
- \[14\]K\. Huang, S\. Zhang, H\. Wang, Y\. Qu, Y\. Lu, Y\. Roohani, R\. Li, L\. Qiu, J\. Zhang, Y\. Di,et al\.\(2025\)Biomni: a general\-purpose biomedical AI agent\.bioRxiv\.External Links:[Document](https://dx.doi.org/10.1101/2025.05.30.656746)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1)\.
- \[15\]S\. Hunter, R\. Apweiler, T\. K\. Attwood, A\. Bairoch, A\. Bateman, D\. Binns, P\. Bork, U\. Das, L\. Daugherty, L\. Duquenne,et al\.\(2009\)InterPro: the integrative protein signature database\.Nucleic Acids Research37\(Suppl\. 1\),pp\. D211–D215\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkn785)Cited by:[§C\.13](https://arxiv.org/html/2609.26836#A3.SS13.p1.1)\.
- \[16\]Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. Bang, A\. Madotto, and P\. Fung\(2023\)Survey of hallucination in natural language generation\.ACM Computing Surveys55\(12\),pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1145/3571730)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px3.p1.1)\.
- \[17\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1)\.
- \[18\]M\. Johns, T\. Meurers, F\. N\. Wirth, A\. C\. Haber, A\. Müller, M\. Halilovic, F\. Balzer, and F\. Prasser\(2023\)Data provenance in biomedical research: scoping review\.Journal of Medical Internet Research25,pp\. e42289\.External Links:[Document](https://dx.doi.org/10.2196/42289)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px2.p2.1)\.
- \[19\]M\. Kanehisa and S\. Goto\(2000\)KEGG: kyoto encyclopedia of genes and genomes\.Nucleic Acids Research28\(1\),pp\. 27–30\.External Links:[Document](https://dx.doi.org/10.1093/nar/28.1.27)Cited by:[§C\.14](https://arxiv.org/html/2609.26836#A3.SS14.p1.1)\.
- \[20\]Z\. A\. King, J\. Lu, A\. Dräger, P\. Miller, S\. Federowicz, J\. A\. Lerman, A\. Ebrahim, B\. O\. Palsson, and N\. E\. Lewis\(2016\)BiGG models: a platform for integrating, standardizing and sharing genome\-scale models\.Nucleic Acids Research44\(D1\),pp\. D515–D522\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkv1049)Cited by:[§C\.8](https://arxiv.org/html/2609.26836#A3.SS8.p1.1)\.
- \[21\]M\. J\. Landrum, J\. M\. Lee, G\. R\. Riley, W\. Jang, W\. S\. Rubinstein, D\. M\. Church, and D\. R\. Maglott\(2014\)ClinVar: public archive of relationships among sequence variation and human phenotype\.Nucleic Acids Research42\(D1\),pp\. D980–D985\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkt1113)Cited by:[§C\.11](https://arxiv.org/html/2609.26836#A3.SS11.p1.1)\.
- \[22\]A\. Lercher, J\. Glock, C\. Macho, and M\. Pinzger\(2024\)Microservice API evolution in practice: a study on strategies and challenges\.Journal of Systems and Software215,pp\. 112110\.External Links:[Document](https://dx.doi.org/10.1016/j.jss.2024.112110)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px1.p1.1)\.
- \[23\]X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. Tang\(2024\)AgentBench: evaluating LLMs as agents\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=zAdUB0aCTQ)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px2.p1.1)\.
- \[24\]P\. Madrigal, A\. S\. Thanki, S\. Fexova, I\. D\. Yu, A\. Chatzigeorgiou, I\. Zucchi, J\. C\. Marugan Calles, L\. Vilmovsky, A\. Khen, L\. Zhao,et al\.\(2026\)Expression atlas in 2026: enabling FAIR and open expression data through community collaboration and integration\.Nucleic Acids Research54\(D1\),pp\. D147–D157\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkaf1238)Cited by:[§C\.12](https://arxiv.org/html/2609.26836#A3.SS12.p1.1)\.
- \[25\]P\. Missier, S\. Woodman, H\. Hiden, and P\. Watson\(2014\)Provenance and data differencing for workflow reproducibility analysis\.External Links:1406\.0905,[Document](https://dx.doi.org/10.48550/arXiv.1406.0905)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px3.p1.1)\.
- \[26\]B\. Mo\(2026\)The age of AI agents demands a new scientific paradigm to sustain trustworthy science\.External Links:2607\.26064,[Document](https://dx.doi.org/10.48550/arXiv.2607.26064)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1)\.
- \[27\]National Library of Medicine\(n\.d\.\)ClinicalTrials\.gov\.Note:WebsiteAccessed 2026\-09\-07External Links:[Link](https://clinicaltrials.gov/)Cited by:[§C\.9](https://arxiv.org/html/2609.26836#A3.SS9.p1.1)\.
- \[28\]R\. D\. Olson, R\. Assaf, T\. Brettin, N\. Conrad, C\. Cucinell, J\. J\. Davis, D\. M\. Dempsey, A\. Dickerman, E\. M\. Dietrich, R\. W\. Kenyon,et al\.\(2023\)Introducing the bacterial and viral bioinformatics resource center \(BV\-BRC\): a resource combining PATRIC, IRD and ViPR\.Nucleic Acids Research51\(D1\),pp\. D678–D689\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkac1003)Cited by:[§C\.2](https://arxiv.org/html/2609.26836#A3.SS2.p1.1)\.
- \[29\]R\. Parasuraman and V\. Riley\(1997\)Humans and automation: use, misuse, disuse, abuse\.Human Factors39\(2\),pp\. 230–253\.External Links:[Document](https://dx.doi.org/10.1518/001872097778543886)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px3.p1.1)\.
- \[30\]Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. Sun\(2024\)ToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=dHng2O0Jjr)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px2.p1.1)\.
- \[31\]C\. D\. Santos\-Júnior, M\. D\. T\. Torres, Y\. Duan, Á\. Rodríguez Del Río, T\. S\. B\. Schmidt, H\. Chong, A\. Fullam, M\. Kuhn, C\. Zhu, A\. Houseman, J\. Somborski, A\. Vines, X\. Zhao, P\. Bork, J\. Huerta\-Cepas, C\. de la Fuente\-Nunez, and L\. P\. Coelho\(2024\)Discovery of antimicrobial peptides in the global microbiome with machine learning\.Cell187\(14\),pp\. 3761–3778\.e16\.External Links:[Document](https://dx.doi.org/10.1016/j.cell.2024.05.013)Cited by:[§C\.1](https://arxiv.org/html/2609.26836#A3.SS1.p1.1)\.
- \[32\]Y\. L\. Simmhan, B\. Plale, and D\. Gannon\(2005\)A survey of data provenance in e\-science\.ACM SIGMOD Record34\(3\),pp\. 31–36\.External Links:[Document](https://dx.doi.org/10.1145/1084805.1084812)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px3.p1.1)\.
- \[33\]S\. M\. Sohan, C\. Anslow, and F\. Maurer\(2015\)A case study of web API evolution\.In2015 IEEE World Congress on Services,pp\. 245–252\.Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px1.p1.1)\.
- \[34\]H\. Soni\(2026\)ToolFailBench: diagnosing tool\-use failures in LLM agents\.External Links:2607\.04686,[Document](https://dx.doi.org/10.48550/arXiv.2607.04686)Cited by:[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px2.p1.1)\.
- \[35\]K\. Swanson, W\. Wu, N\. L\. Bulaong, J\. E\. Pak, and J\. Zou\(2025\)The virtual lab of AI agents designs new SARS\-CoV\-2 nanobodies\.Nature646\(8085\),pp\. 716–723\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09442-9)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1)\.
- \[36\]A\. Taktakidze\(2026\)A new paradigm: agentic AI for scientific discovery\.Longevity Horizon2\(5\)\.External Links:[Document](https://dx.doi.org/10.65649/ukh4n33x)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1)\.
- \[37\]The wwPDB Consortium\(2024\)EMDB—the electron microscopy data bank\.Nucleic Acids Research52\(D1\),pp\. D456–D465\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkad1019)Cited by:[§C\.7](https://arxiv.org/html/2609.26836#A3.SS7.p1.1)\.
- \[38\]K\. Vukojević, N\. Alfirević, P\. Todorović, and M\. Maglica\(2026\)Cheap hypotheses, expensive truths: science in the age of generative AI\.SSRN Electronic Journal\.External Links:[Document](https://dx.doi.org/10.2139/ssrn.6352998)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1)\.
- \[39\]M\. D\. Wilkinson, M\. Dumontier, I\. J\. Aalbersberg, G\. Appleton, M\. Axton, A\. Baak, N\. Blomberg, J\. Boiten, L\. B\. da Silva Santos, P\. E\. Bourne,et al\.\(2016\)The FAIR guiding principles for scientific data management and stewardship\.Scientific Data3,pp\. 160018\.External Links:[Document](https://dx.doi.org/10.1038/sdata.2016.18)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.26836#S5.SS0.SSS0.Px2.p2.1)\.
- \[40\]K\. Zhu, Z\. Liu, B\. Li, M\. Tian, Y\. Yang, J\. Zhang, P\. Han, Q\. Xie, F\. Cui, W\. Zhang, X\. Ma, X\. Yu, G\. Ramesh, J\. Wu, Z\. Liu, P\. Lu, J\. Zou, and J\. You\(2025\)Where LLM agents fail and how they can learn from failures\.External Links:2509\.25370,[Document](https://dx.doi.org/10.48550/arXiv.2509.25370)Cited by:[§1](https://arxiv.org/html/2609.26836#S1.SS0.SSS0.Px1.p1.1)\.
- \[41\]Z\. Zou, T\. Ohta, and S\. Oki\(2024\)ChIP\-Atlas 3\.0: a data\-mining suite to explore chromosome architecture together with large\-scale regulome data\.Nucleic Acids Research52\(W1\),pp\. W45–W53\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkae358)Cited by:[§C\.5](https://arxiv.org/html/2609.26836#A3.SS5.p1.1)\.

## Appendix AFailure Measurement

Table 1:Dimensional classification of failures and rubrics\.MeasureDimensionalityKey checks / verification requirements1\. CompletenessInformation, field, record, and qualifier coverageCheck whether all requested information is present; whether fields, records, references, qualifiers, alternatives, or pagination have been omitted; whether a response that claims completeness is actually complete; compare API/tool output→\\rightarrowwrapper output→\\rightarrowagent response\.2\. CorrectnessFactual, identifier, logical, and semantic accuracyVerify values against authoritative sources; check identifiers, mappings, calculations, and claims; check whether the agent confused entities or interpreted a tool field incorrectly; distinguish retrieved facts from inferred facts\.3\. AppropriatenessTool, endpoint, and parameter selection; query construction; scope fitCheck whether the correct tool/endpoint was selected; whether the tool can answer the question; whether parameters and syntax are appropriate; whether a narrow tool was incorrectly used as a comprehensive source; whether fallback tools change the intended scope\.4\. ConsistencyRepeatability; cross\-query, cross\-source, and temporal consistencyRun equivalent queries repeatedly; compare alternative formulations and reverse\-direction queries; test ordering, batching, and namespace transformations; determine whether differences are explainable by source semantics or execution conditions\.5\. ReliabilityExecution stability, error handling, recovery, and dependency behaviorCheck for timeouts, errors, empty results, truncation, and partial responses; determine whether failures are surfaced; test whether retries/fallbacks preserve scope; check whether apparent success represents failed or partial execution\.6\. AttributionSource and evidence provenance, citation completeness, and traceabilityCheck whether claims can be traced to sources; whether identifiers, references, evidence codes, and source records survive; whether the tool→\\rightarrowwrapper→\\rightarrowagent chain can be reconstructed; whether citations support the specific claims\.7\. InterpretationSource neutrality, ranking effects, hidden assumptions, and confidence/calibrationCheck whether ranking or selection logic is exposed; whether defaults or hidden weighting affect results; whether ranked results become factual conclusions; whether uncertainty, confidence, or evidence strength is preserved; check for subjective framing\.8\. RelevanceSemantic meaning, qualifier preservation, contextual interpretation, and inferenceCheck whether fields and symbols are correctly interpreted; whether qualifiers such as evidence, status, species, confidence, direction, temporal state, or experimental/computational origin survive; whether qualified claims become absolute; distinguish source content from agent inference\.9\. Context & conditionsQuery alignment, information relevance, scope relevance, and downstream relevanceCheck whether the response answers the question asked; whether relevant records are included; whether irrelevant information replaces requested information; whether the answer remains at the requested specificity; whether narrowing or substitution is disclosed\.10\. Purpose alignmentIntended\-use alignment, contextual fitness, downstream impact, and purpose evolutionCheck whether the output suits the intended use; whether tool limitations affect that use; whether a technically valid answer could produce a misleading downstream conclusion; check changes in tool/API purpose or scope where relevant\.
## Appendix BDimensions, Failure Loci, and Representative Instances

Table 2:Dimensions, associated failure loci, and representative instances\.DimensionAssociated Failure Loci \(indicative\)InstancesCompletenessL2, L1, L7, L51\. BiGG Models \(L2, BIGG\_TC\_13\): Advanced search model formats exists in the Web UI but has no corresponding API endpoint\.2\. ChIP\-Atlas \(L1, CHIPATLAS\_TC\_02\): "Diff Analysis" is listed but disabled at the tool level; DMR functionality formally removed\.3\. ClinicalTrials\.gov \(L4, CTG\_TC\_02\): "More Search Options" \(FDAAA 801 violations, results\-submitted states\) undocumented as wrapper\-accessible parameters\.4\. AMPSphere \(L5, AMPS\_TC\_01\): Helical\-wheel projection generation has no API operation at all — absent at the tool level, not just the wrapper\.Interpretation VariabilitiesL7, L3, L1, L51\. KEGG \(L7, KEGG\_TC\_15\): Agent correctly parses OR\-logic within a module group but misapplies the AND\-completeness rule across blocks\.2\. BiGG Models \(L3, BIGG\_TC\_08\): Multiple charges/formulae returned for one metabolite ID with no indication of which is preferred\.3\. ChIP\-Atlas \(L1, CHIPATLAS\_TC\_09\): The Peak Browser contains 455 tracks labeled "Unclassified" cell type class, creating variability in how an agent should treat these samples in tissue\-specific meta\-analyses\.4\. BV\-BRC \(L5, BVBRC\_TC\_4\): API returns the critical quality metrics \(’checkm\_completeness’, ’checkm\_contamination’, ’genome\_quality’\), as confirmed by the tool\_api\_response\. ’BVBRC\_get\_genome’ wrapper fails to include these fields in its return schema – omits them from the actual data returned to the agent\.Result AppropriatenessL7, L2, L3, L51\. ChIP\-Atlas \(L7, CHIPATLAS\_TC\_08\): Agent misreads MACS2’s \-10×\\timeslog10\(Q\-value\) score as a raw/linear magnitude\.2\. AMPSphere \(L2, AMPS\_TC\_04\): Undocumented whether exact\-match search surfaces homology\-based hits from the separate MMseqs/HMMER endpoints\.3\. BiGG Models \(L3, BIGG\_TC\_07\): Search results silently capped with no documented page size\.4\. BiGG Models \(L5, BIGG\_TC\_14\): Gene detail API returns placeholder values \(e\.g\., single\-space protein sequence\) that the wrapper passes through unflagged\.Source and Data AttributionL3, L1, L51\. BiGG Models \(L3, BIGG\_TC\_04\): PMID reference fields exist in schema but are empty for specific named models\.2\. AMPSphere \(L1, AMPS\_TC\_08\): "Progenomes" — a genome database source, not an environmental location — is incorrectly classified as a habitat value alongside genuine habitats like "human gut\."3\. BV\-BRC \(L5, BVBRC\_TC\_03\): Genome metadata mixes internal, legacy, and external NCBI identifiers with no authoritative\-source flag\.Ranking and RelevanceL5, L3, L41\. AMPSphere \(L5, AMPS\_TC\_04\): MMseqs2/HMMER endpoints exist but ranking metric \(E\-value/Bit score/Identity\) is undocumented\.2\. BiGG Models \(L3, BIGG\_TC\_07\): Exact vs\. prefix vs\. substring matching logic undocumented at the API level\.3\. ClinicalTrials\.gov \(L4, CTG\_TC\_10\): Web UI defaults to "Relevance" sort; API default sort order differs/is unclear\.Reliability of InformationL5, L31\. BiGG Models \(L5, BIGG\_TC\_08\): Documentation reports API v1\.3\.0 while live responses report v1\.6\.0 — confirmed schema/content drift\.2\. Expression Atlas \(L3, GXA\_TC\_05\): Accession\-naming drift between E\-MTAB and E\-GTEX conventions over time creates evolutionary reliability risk for long\-running studies \(this is the same E\-MTAB\-5214/E\-GTEX\-8 pair we independently verified earlier via live API\)\.3\. BV\-BRC \(L5, BVBRC\_TC\_12\): Legacy PATRIC\-era p2\_genome\_id retained for cross\-walks may drift from current BV\-BRC taxonomy\.
## Appendix CTool\-wise representative examples

### C\.1AMPSphere

AMPSphere provides antimicrobial\-peptide sequence and functional information\[[31](https://arxiv.org/html/2609.26836#bib.bib16)\], but failures arise from ambiguous biological semantics, incomplete contextual information, and difficulties in using or interpreting returned results\. The key failures observed are:L1:The ‘empty’ and ‘others’ microbial\-source categories do not provide sufficiently specific semantic definitions to support environmental interpretation\.L3:The API exposes individual AMP properties but not equivalent comparative context; for example, a GRAVY value such as−0\.11\-0\.11is returned without the family distribution needed to determine whether it is unusual\.L7:The agent transformed multiple QC indicators into a categorical conclusioN\.

### C\.2BV\-BRC

BV\-BRC provides microbial genome information whose interpretation depends on genome quality, provenance, identifier semantics, and reference status\[[28](https://arxiv.org/html/2609.26836#bib.bib17)\]\. The key failures observed are:L5:The genome collection contains heterogeneous evidence and quality levels, including MAGs and finished or high\-quality draft genomes\. The returned records do not establish these genome types as equivalent biological references\.L3:Identifier and error behavior can be ambiguous; an invalid or obsolete identifier may produce an empty result without clearly distinguishing ‘no record’ from ‘retrieval failure\.’L5:The reference/representative genome was missing from the wrapper, the agent might select one as the reference without source evidence establishing that designation\.

### C\.3ClinGen

ClinGen provides curated genomic validity information\[[4](https://arxiv.org/html/2609.26836#bib.bib18)\], but mismatches in API functionality, output representation, and biological granularity can lead agents to answer the wrong scientific question\. The key failures observed are:L2:Drug\-name/RXNORM searching available through the Web interface is not represented as a corresponding API capability; for example, a drug\-oriented search such as “aspirin” cannot be reproduced through the documented validity endpoints\.L3:Returned information may not be directly usable for the requested analytical operation when the available representation does not match the expected input or output structure\.L7:The agent returned a successful response from a related endpoint as the answer to the requested question, although the relevant variant\-level evidence was not retrieved\.

### C\.4CIViC

CIViC integrates clinical variant evidence and assertions, making evidence granularity, search coverage, and evidence\-count interpretation critical for reliable agent use\[[12](https://arxiv.org/html/2609.26836#bib.bib19)\]\. The key failures observed are:L5:The API can flatten distinctions between individual Evidence items and synthesized Assertions, reducing the granularity of the underlying clinical evidence\.L5:Advanced Web search supports Boolean logic and ontology expansion that are not equivalently exposed through the API; for example, a parent disease search may not automatically retrieve evidence associated with specific subtypes\.

### C\.5ChIP\-Atlas

ChIP\-Atlas provides large\-scale epigenomic information, but biological\-state normalization, score semantics, ranking, and result retrieval introduce multiple failure loci\[[41](https://arxiv.org/html/2609.26836#bib.bib20)\]\. The key failures observed are:L1:Biologically meaningful modifications may be intentionally collapsed; for example, phospho\-SMAD3 is normalized to SMAD3, removing information about the phosphorylation state\.L2:Enrichment analysis can return a submission URL rather than structured results, preventing a fully programmatic retrieval workflow\.L3:Search and ranking semantics are incompletely specified; for example, FTS5 returns ranked results without clearly defining the metadata contributing to relevance\.L7:The agent interpreted a MACS2 score of 500 as a raw count rather than the documented−10​log10​\(q\)\-10\\log\_\{10\}\(q\)transformation\.L7:The agent treated “Unclassified” tracks as unusable and consequently excluded potentially relevant records from the retrieved set\.

### C\.6CryoET Data Portal

CryoET Data Portal provides cryo\-electron tomography datasets\[[8](https://arxiv.org/html/2609.26836#bib.bib21)\], but provenance, attribution, and retrieval metadata may not survive consistently across the automated interface\. The key failures observed are:L3:Publication metadata may be returned as null; for example,datasetPublicationscan be absent despite publication context associated with the dataset\.L5:The official GraphQL API supports retrieving child datasets for a deposition \(evidenced by the API response which successfully returns datasets linked to deposition 10014\), but the CryoET\_list\_depositions wrapper completely omits this field from its return schema, preventing the agent from discovering the hierarchical relationship\.

### C\.7EMDB

EMDB provides structural biology records through identifiers and searchable metadata\[[37](https://arxiv.org/html/2609.26836#bib.bib22)\], but retrieval limits and identifier representations can produce failures that propagate into agent interpretation\. The key failures observed are:L5:Search results are capped; for example, therowsparameter is limited to 1000 despite the database containing substantially more records, creating systematic truncation for broad searches\.L3:The official documentation for the tool’s advanced search interface specifies dozens of filterable fields, but the actual implementation of the REST API \(as verified by the failure of the fielded queries in the baseline test and the documentation’s limitation to a simple keyword search endpoint\) only supports basic keyword searching\. This constitutes a partial functional gap where the API fails to provide the same capability as the WebUI\.

### C\.8BiGG Models

BiGG Models provides metabolic models and associated metadata\[[20](https://arxiv.org/html/2609.26836#bib.bib23)\], but differences between Web, API, and wrapper capabilities can constrain automated model discovery\. The key failures observed are:L3:Model metadata can differ between search and model\-detail representations; for example, organism information available in the search context may be absent from the model detail response\.L3:The Web interface supports more advanced multi\-field search than the corresponding REST API\.L5:The official Tool API successfully returns gene sequence and strand data, but the ToolUniverse wrappers do not expose these fields to the agent\. While the BiGG\_get\_model wrapper exists, it only provides high\-level metadata; there is no wrapper functionality to access specific gene\-level details returned by the underlying API endpoint \(e\.g\., /api/v2/models/\{model\_id\}/genes/\{gene\_id\}\), forcing the agent to fail\.

### C\.9ClinicalTrials\.gov

ClinicalTrials\.gov provides extensive clinical\-trial search and eligibility information\[[27](https://arxiv.org/html/2609.26836#bib.bib24)\], but API filtering limitations can change the population retrieved by an autonomous agent\. The key failures observed are:L3:The wrapper returns a success status with empty outcomes but no explicit ’No Results Posted’ flag\. This is a silent failure to convey the definitive ’No Results Posted’ state clearly shown in the Web UI\.L7:Agent was asked to find studies which had "FDAAA 801 Violations"\. Instead of using the correct query term which is "FDAAA 801" only, it used the entire string "FDAAA 801 violations" to search, due to which the query returned empty response

### C\.10CTIS

CTIS provides clinical\-trial information\[[9](https://arxiv.org/html/2609.26836#bib.bib25)\]through Web and API interfaces whose search semantics, status representations, and result\-shaping capabilities are not fully aligned\. The key failures observed are:L3:Web “Display options” for pagination and column selection do not have equivalent API controls, limiting reproducible result shaping\.L7:Wrapper returned breast\-cancer trials for limit=100 but agent reports no total\_pages/next\_link pagination metadata\. This includes lacking pagination metadata\.L5:The agent treated API search behavior as equivalent to the Web\-interface Boolean filtering semantics, although the API did not establish equivalent behavior\.

### C\.11ClinVar

ClinVar contains submitted reports of human genomic variants for diseases and drug responses with supporting evidence\[[21](https://arxiv.org/html/2609.26836#bib.bib26)\], however, there are few API/wrapper differences that might cause misinterpretations in agents\. For example, L5: Detailed submission\-level observation metadata, including Age, Sex, Clinical Features, and Collection Method, is present in the underlying ‘efetch‘ XML but omitted by the ToolUniverse wrapper, L5: The Web interface distinguishes submissions that contribute to the aggregate classification from those that do not using ‘C‘/‘N‘ badges, but this contribution status is not exposed by the wrapper\. L7: Consequently, the agent inferred contribution status from the submitted classifications rather than using the explicit contribution attribute, potentially misrepresenting the basis and strength of the aggregate classification\.

### C\.12Expression Atlas

Expression Atlas is an open source resource providing gene and protein expression data for different biological conditions\[[24](https://arxiv.org/html/2609.26836#bib.bib27)\]\. Some inconsistencies and incompleteness in information through API/wrapper might hinder agentic workflows\. L3: The Expression Atlas Web UI provides gene\-level expression values \(TPM/FPKM\), but the documented API exposes only experiment\-level metadata and no equivalent programmatic endpoint for retrieving the full expression data\. L3: The Web UI exposes curated collections such as ENCODE and BLUEPRINT, but the API’s ‘experimentProjects‘ field can be empty even for experiments belonging to these projects, preventing reliable programmatic identification of project affiliations\. L3: Experiment accession identifiers are inconsistent across the Web interface and API \(e\.g\., E\-MTAB\-5214 vs E\-GTEX\-8\), and the ToolUniverse tool fails to resolve the former or provide alias mapping\. L3: The Web UI provides publication/citation metadata, but the experiment API omits PubMed IDs and other citation fields\. L3: Expression values lack explicit TPM/FPKM unit metadata in the API, requiring the agent to infer the units rather than receiving them as a qualifier\.

### C\.13InterPro

Interpro provides functional analyses of proteins by classifying protein sequences into families and predicting the presence of important sites\[[15](https://arxiv.org/html/2609.26836#bib.bib28)\], in which certain differences in features and capabilities are partially or completely missing in the API/wrapper\. L3: The InterPro Web interface provides a ‘Search by sequence” capability powered by InterProScan, while the official InterPro REST API does not have a similar endpoint for submitting raw protein sequences for scanning, preventing agents from performing on\-the\-fly domain classification through the core API\. L3: The Web interface supports ‘Search by Domain Architecture” to identify proteins matching ordered combinations of domains, while the API does not provide an equivalent database\-wide architecture\-pattern search, limiting automated discovery of proteins with specific domain arrangements\. L3: The Web interface displays full taxonomic lineage on protein pages, whereas the API response exposes limited organism information and requires additional taxonomy requests to reconstruct the lineage available directly to Web users\.

### C\.14KEGG

The KEGG database and analysis tools enable understanding high\-level functionalities of biological systems including cells, organisms, ecosystems from molecular information\[[19](https://arxiv.org/html/2609.26836#bib.bib29)\]\. However absence of certain functionalities restrict agentic workflows from utlizing the complete capabilities of KEGG\. L2: High\-level analytical operations available on the Web \(KEGG Mapper’s Reconstruct/Color/Join, and sequence\-based functional annotation via BlastKOALA/GhostKOALA\) are not exposed through the REST API, which offers only raw lookup, list, and conversion operations\. L2: The API cannot reproduce the full analytical and annotation capability of the Web interface programmatically, and where richer machine\-readable structure does exist \(e\.g\. KGML\), it is returned as unparsed, interactivity\-stripped XML rather than an agent\-consumable format\. L7: The agent compensated for the absence of a computed module\-completeness endpoint by manually interpreting the DEFINITION field’s boolean syntax itself, correctly identifying comma\-separated K\-numbers as alternative \(OR\) options but incorrectly generalizing this to conclude that a missing K\-number within a group never compromises module completeness\.

### C\.15Reactome

Reactome provides peer\-reviewed pathway information\[[5](https://arxiv.org/html/2609.26836#bib.bib30)\]through Web and API interfaces whose hierarchical relationships, identifier semantics, search behaviour, and analysis metadata are not fully aligned\. The key failures observed are: L2: The Reactome Content Service does not provide a per\-instance ‘referrers” endpoint or equivalent field, requiring agents to reconstruct parent relationships through child ‘eventOf” links and potentially perform N\+1 queries, limiting complete traversal of the pathway hierarchy\. L3: Species identifiers are inconsistent across Reactome services where Content Service operations may use NCBI taxonomy IDs such as ‘9606”, whereas Analysis Service operations such as species comparison require Reactome\-specific ‘dbId” values such as ‘48892”, forcing agents to translate between two identifier systems for the same species\. L3: Reactome search exposes clustering as a default behaviour while the corresponding ToolUniverse search interface does not expose a ‘cluster” parameter, and the API enforces a 200\-character limit on the ‘q” parameter, restricting complex automated searches\. L7: Reactome’s use of ‘Reaction” encompasses a broad range of biological state changes, including binding and translocation events, which may lead agents to interpret Reactome reaction records as conventional biochemical conversions when they are not\. L3: The Content and Analysis services expose different service\-version identifiers, creating additional uncertainty about whether outputs generated across the two interfaces correspond to the same API release\.