Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools

# Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools
Source: [https://arxiv.org/html/2608.04719](https://arxiv.org/html/2608.04719)
###### Abstract

Agent evaluations tell us*that*a model picked the wrong tool, but rarely*why*\. We introducecanary tools: diagnostic probe tools planted in an agent’s Model Context Protocol \(MCP\) tool set, each engineered to probe one specific tool\-selection weakness\. A six\-type taxonomy \(semantic decoys, parameter traps, capability mirages, prerequisite blindness, temporal decoys, and granularity traps\) turns a single “wrong tool” outcome into a multi\-dimensional profile of how a model reasons about tools\. We evaluate eight models, six hosted models from three providers plus two 8B open\-weight models, spanning three capability tiers, on 120 tasks across three canary\-density conditions and three seeds \(8,640 task runs\), together with a 2,880\-run controlled subtlety ablation\. Task success is graded by a provider\-independent judge, one that is neither among the tested models nor shares a provider with any of them, and an independent second judge corroborates it \(inter\-judge Cohen’sκ=0\.75\\kappa\\\!=\\\!0\.75\)\. We report three findings\. First, susceptibility drops sharply as models get more capable: the per\-task canary susceptibility rate \(csr\) ranges about36×36\\timesacross the eight models, lowest for Claude Opus 4\.8 and highest for Llama 3\.1 8B\. Second,*capability tier alone does not predict safety*: the most susceptible of the six hosted models is mid\-tier, and within a provider the cheaper model can be the safer one\. Third, the taxonomy is*capability\-stratified*: capability mirages are the probe that most reliably traps frontier models, while the remaining types are largely inert on strong models but fire readily on the small open models, so they discriminate by capability rather than being weak\. Softening each canary’s give\-away phrase leaves frontiercsressentially unchanged, evidence that the probes measure reasoning rather than phrase\-spotting\. Susceptibility also predicts task failure \(Spearmanρ=−0\.34\\rho\\\!=\\\!\-0\.34\), while the most robust models are not significantly degraded by canary pressure\. We release the framework, canary schemas, tasks, and logs\.

Diagnosing Tool\-Selection Reasoning in LLM Agents with Canary Tools

Atul Anand Sourav Chattaraj

## 1Introduction

Agent benchmarks tell us*whether*an LLM agent completed a task, but rarely*why*it failed when it did\(Liu et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib5); Qin et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib9)\)\. Tool\-using agents\(Schick et al\.,[2023](https://arxiv.org/html/2608.04719#bib.bib10); Yao et al\.,[2023](https://arxiv.org/html/2608.04719#bib.bib12); Patil et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib8)\)increasingly work over large, heterogeneous tool sets exposed through standards such as the Model Context Protocol\(Anthropic,[2024](https://arxiv.org/html/2608.04719#bib.bib1)\), and a recurring failure mode is*tool selection*: shown two similar tools, the agent reaches for the one that looks right by name but is wrong in function\. Distractor\-based suites add irrelevant tools and check whether performance drops, but they still return a single bit, that the agent failed, not*which reasoning step*broke\. For improving an agent that is the wrong level of detail: a developer who wants to fix a tool\-selection failure first needs to know what*kind*of mistake it was \(Figure[1](https://arxiv.org/html/2608.04719#S1.F1)\)\.

Standard evaluation\.agent picks a tool→\\rightarrow*did it fail?*→\\rightarrowyes / no \(one bit\)Canary diagnosis \(this work\)\.the*same*wrong pick answers*which reasoning step broke?*semanticparametercapabilityprerequisitetemporalgranularitytypedFigure 1:The diagnostic shift\. A standard benchmark reduces a tool\-selection error to one bit \(fail or not\)\. A canary is engineered so that the same wrong pick identifies*which*of six reasoning weaknesses occurred, turning the outcome into a typed diagnosis\.We argue for a diagnostic shift, borrowing an idea from misconception probes in educational testing: instead of scoring an answer right or wrong, design each item so that a particular wrong answer reveals a particular misconception\. We bring this idea to tool selection withcanary tools, diagnostic probe tools planted in the agent’s tool set, each built so that calling it exposes one specific reasoning weakness\. A canary is not noise\. It is a targeted probe with known failure semantics\.

Our contributions are:

1. 1\.A six\-typecanary taxonomy\(§[3](https://arxiv.org/html/2608.04719#S3)\), each type probing a distinct tool\-selection capability\.
2. 2\.A reproduciblegeneration\-and\-evaluation framework\(§[4](https://arxiv.org/html/2608.04719#S4)\): a schema\-driven canary generator, a realistic sandboxed tool environment, an agent loop spanning hosted and local models, a provider\-independent outcome judge, and a trap detector\.
3. 3\.An empirical study ofeight models\(§[6](https://arxiv.org/html/2608.04719#S6)\) yielding a capability\-stratified diagnostic profile, the finding that capability tier does not predict safety, and validation that canary susceptibility predicts task failure\.

Figure[2](https://arxiv.org/html/2608.04719#S1.F2)previews the headline result: the per\-task canary susceptibility rate \(csr\) spans roughly36×36\\timesacross models, and capability tier does not order them \(the most susceptible model is mid\-tier, and a mid\-tier model beats its same\-provider frontier sibling\)\.

![Refer to caption](https://arxiv.org/html/2608.04719v1/figures/ctr_by_model.png)Figure 2:Per\-task canary susceptibility rate per model \(declared condition; lower is better\)\. Bars are ordered by provider group, not by capability tier\. Tier does not order the models: the worst is mid\-tier, and a mid\-tier model beats its frontier sibling\. Full numbers with confidence intervals in Table[1](https://arxiv.org/html/2608.04719#S6.T1)\.
## 2Related Work

#### Agent and tool\-use benchmarks\.

Most agent and tool\-use benchmarks score whether the call or the task was correct\. AgentBench\(Liu et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib5)\)and ToolLLM\(Qin et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib9)\)measure end\-to\-end task completion over large tool sets; the Berkeley Function\-Calling Leaderboard\(Patil et al\.,[2025](https://arxiv.org/html/2608.04719#bib.bib7)\)checks whether a function call matches a reference via abstract\-syntax\-tree comparison; andτ\\tau\-bench\(Yao et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib11)\)and ToolSandbox\(Lu et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib6)\)evaluate multi\-turn, stateful tool use against a policy or milestone checks\. All are outcome\- or correctness\-based: a wrong selection is scored incorrect, but none reports*which*reasoning weakness produced it\. Canary tools are complementary, attaching a typed diagnosis to the wrong pick\.

#### Distractors and tool selection\.

A closer line of work adds extra tools to probe selection directly\. MetaTool\(Huang et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib3)\)tests whether to call a tool and which one, including selection among similar tools and tools with stated reliability issues\. MCPAgentBench\(Liu et al\.,[2025](https://arxiv.org/html/2608.04719#bib.bib4)\)and MCP\-Atlas\(Bandi et al\.,[2026](https://arxiv.org/html/2608.04719#bib.bib2)\)place irrelevant or similar\-domain tools alongside the correct one to measure anti\-interference\. These suites yield a binary signal \(did the agent pick a distractor?\)\. A canary, by contrast, is engineered so that*which*canary the agent takes reveals a*specific*reasoning weakness, which turns one wrong pick into a multi\-dimensional profile rather than a single interference score\.

#### Tool hallucination and security probes\.

ToolBeHonest\(Zhang et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib14)\)diagnoses hallucination in tool\-augmented models, such as invoking non\-existent tools or arguments\. That failure originates in the model’s weights, whereas we study selection among plausible, real\-looking tools presented in context\. On the security side, InjecAgent\(Zhan et al\.,[2024](https://arxiv.org/html/2608.04719#bib.bib13)\)injects adversarial content to test indirect prompt\-injection robustness, and canary tokens are a classic honeypot for detecting intrusion\. We borrow the honeypot intuition but aim it at benign tool\-selection reasoning rather than at adversarial attacks\.

## 3The Canary Taxonomy

#### Problem setup\.

An agent is given a taskqqand a tool setTT, where each toolttexposes a name, a natural\-language description, and a parameter schema\.*Tool selection*is the choicea​\(q,T\)∈Ta\(q,T\)\\in Tof which tool to call\. A correct choice requires the agent’s decision to be*sensitive*to the properties ofttthat determine its fitness forqq: what the tool actually does \(not merely what its name suggests\), whether its inputs can be satisfied, whether its advertised capabilities are real and needed, whether it has unmet prerequisites, whether its data is current, and whether its scope matches the task\. Acanaryis a targeted perturbation of a real toolt⋆t^\{\\star\}along exactly one of these axes,c=δ​\(t⋆\)c=\\delta\(t^\{\\star\}\), that leaves the tool superficially plausible but wrong forqq\. PlantingccinTTand observinga​\(q,T∪\{c\}\)=ca\(q,T\\cup\\\{c\\\}\)=cshows the agent’s selection was*insensitive*to the perturbed property, which is a typed failure rather than a generic “wrong tool\.” The taxonomy enumerates one canary type per axis\. The six axes cover the components of a tool’s contract \(name, description, parameter schema, and scope\); we do not claim they are exhaustive, and the framework admits further axes\. Their non\-redundancy is also visible empirically: each type fires at a different point on the capability curve \(§[6](https://arxiv.org/html/2608.04719#S6)\)\.

Each canary type therefore targets one reasoning capability and has a deterministic failure semantics, so that a trapped call is unambiguous\.

Semantic decoy\.A tool with a paraphrased name and description but identical schema, whose description reveals it returns cached/stale data\. Probes whether the agent reads descriptions or name\-matches\.

Parameter trap\.A functional look\-alike that renames parameters and adds an unsatisfiable required argument \(e\.g\. anapi\_keythe agent cannot supply\)\. Probes parameter\-feasibility checking\.

Capability mirage\.A variant whose description inflates its powers \(“research\-grade,” “solves the hardest cases”\)\. Probes whether the agent critically evaluates capability claims or prefers the “more powerful” tool\.

Prerequisite blindness\.A privileged variant whose description omits that authentication is required; calling it returns an auth error\. Probes reasoning about implicit prerequisites\.

Temporal decoy\.A variant marked with an outdated date/version\. Probes whether the agent weighs recency\.

Granularity trap\.An over\-specific, hardcoded variant \(e\.g\. weather for one city, no parameters\)\. Probes whether the agent matches tool specificity to task scope\.

#### A worked example\.

Figure[3](https://arxiv.org/html/2608.04719#S3.F3)traces one real trapped run\. On the easy task*“Convert 5 miles to kilometres,”*whose only correct tool isconvert\_units, GPT\-4\.1 instead calls the capability\-mirage canaryadvanced\_convert\_units\. A unit conversion needs none of the advertised “research\-grade” power, so the wrong pick localizes exactly what this probe targets: a preference for the more\-powerful\-sounding tool without checking whether the task warrants it\. This is the diagnostic content a binary distractor score \(“wrong tool”\) would miss\.

Task \(easy\)\.“Convert 5 miles to kilometres\.”Correct tool:convert\_units\(value, from, to\)\.Tool set \(shuffled\)\.12 real tools \+ canaries\. Two are plausible here:∙\\bulletconvert\_units– real converter\.∙\\bulletadvanced\_convert\_units– canary, “optimized, research\-grade\.”Agent \(GPT\-4\.1\)callsadvanced\_convert\_units\.Capability\-mirage trap sprung; the canary returns a value flagged “may be unreliable\.”Diagnosis\.Prefers the more\-powerful\-sounding tool without checking the task needs it\. A distractor score reports only “wrong tool”; the canary names*which*misconception \(capability mirage\)\.Figure 3:A canary probe in action \(real run\)\. The same wrong pick that a distractor benchmark would score as a binary “wrong tool” is, under the canary taxonomy, a typed diagnosis: a*capability mirage*\. Each canary type yields a different such diagnosis\.

## 4Methodology

#### Canary generator\.

Given a real MCP tool schema, the generator emits one canary per type\. Parameter, prerequisite, temporal, and granularity canaries are produced by deterministic schema transforms; semantic and capability canaries additionally use an LLM to reword names and descriptions so they are not trivially distinguishable from the real tool \(per\-type transforms in Appendix[C](https://arxiv.org/html/2608.04719#A3)\)\. The full canary pool is generated once and persisted, so every model, condition, and seed sees byte\-identical canaries\.

#### Realistic tool environment\.

We implement 12 real tools across five MCP servers \(weather, math, file, search, database\)\. Tool results are*realistic synthetic data*: plausible search snippets carrying figures, file contents keyed to filenames, and internally consistent database rows, rather than obvious placeholders\. This matters more than it sounds\. In early runs with placeholder outputs, the stronger models noticed the results were synthetic and simply stopped rather than fabricate an answer, which deflated both their trap rate \(fewer calls, fewer chances to slip\) and their task success\. Realistic outputs let every model work through the multi\-step tasks on equal footing\.

#### Assembler and agent loop\.

For each task, the assembler merges the real tools with the canaries of the declared types and shuffles them deterministically\. Every model, whether hosted \(through an OpenAI\-compatible gateway\) or local \(through Ollama\), runs through the same tool\-calling loop, which removes per\-provider adapter confounds\.

#### Outcome judge\.

Task success \(tsr\) is graded by an LLM judge that reads the task, the tool\-call trace, and the final answer, and decides whether the outcome was achieved\. This replaces a tool\-coverage heuristic \(did the agent call all “correct” tools\), which unfairly penalizes capable models that solve a step by reasoning instead of calling the expected tool \(full prompt in Appendix[B](https://arxiv.org/html/2608.04719#A2)\)\. To avoid grading a model’s output with a member of its own family, the judge is*provider\-independent*: we use DeepSeek\-V3\.2, which is neither one of the eight tested models nor shares a provider with any of them\. We query the judge at temperature 0 so that identical inputs receive identical verdicts, which makestsrreproducible\.

#### Validating the judge\.

We check the judge three ways\. First, is it stable? Re\-grading a 300\-run sample three times at a higher temperature gives verdicts that are unanimous and match the temperature\-0 grade on 94% of runs\. Second, does the choice of judge matter? Re\-grading that same sample with a second independent judge \(GLM\-5, also from outside the tested pool and a different provider\) agrees at Cohen’sκ=0\.75\\kappa\\\!=\\\!0\.75, and the two judges’ overalltsrdiffers by only0\.050\.05\. Third, does it match a person? An author hand\-graded 40 task\-runs and agreed with the judge on 95% \(κ=0\.90\\kappa\\\!=\\\!0\.90\); every disagreement was a case where the judge was the*stricter*of the two, so if anythingtsrunderstates success\. Protocols and cases are in Appendices[D](https://arxiv.org/html/2608.04719#A4)and[E](https://arxiv.org/html/2608.04719#A5)\.

#### Trap detector and metrics\.

The detector classifies every call as real, canary \(with type/id\), or unknown\. We report: per\-taskCanary Susceptibility Rate\(csr\), the mean over tasks of \(canary calls//tool calls\);type\-specific trap rate, the fraction of tasks \(where a type was present\) on which at least one canary of that type was called;Recovery Rate, the fraction of trapped tasks where the agent subsequently called a correct tool; andtsr\. A trap counts as sprung on the first canary call, so an agent that probes a canary, finds it unsatisfiable \(for example a missingapi\_key\), and switches to the real tool is scored as trapped but recovered\. Recovery thus separates this back\-off from the initial mistake\.

## 5Experimental Setup

We evaluate eight models: three frontier \(Claude Opus 4\.8, GPT\-5\.2, Gemini 2\.5 Pro\), three mid\-tier \(Claude Sonnet 4\.6, GPT\-4\.1, Gemini 2\.5 Flash\), and two small open\-weight models \(Llama 3\.1 8B, Qwen3\-8B\)\. Hosted models run through a LiteLLM gateway; the open\-weight models run locally via Ollama \(exact model identifiers in Appendix[A](https://arxiv.org/html/2608.04719#A1)\)\. The suite is 120 tasks \(40 easy, 40 medium, 40 hard\)\. We run three canary\-density conditions,baseline\(no canaries\),declared\(each task’s declared types\), andfull\(all six types\), each with three seeds, for8×3×3×120=8,6408\\times 3\\times 3\\times 120=8\{,\}640task runs\. Baselinecsris0for every model, which confirms the canaries are the only source of trapped calls\. A separate subtlety ablation \(§[6\.7](https://arxiv.org/html/2608.04719#S6.SS7)\) re\-runs the declared condition for all eight models against a softened\-tell canary pool \(8×3×120=2,8808\\times 3\\times 120=2\{,\}880additional runs\)\.

Agents are queried with each provider’s default decoding settings, and the three seeds permute tool ordering rather than fixing a sampling seed, so the reported per\-seed spread reflects both tool\-order and residual sampling variance\. We treat this conservatively: findings we report survive across all three seeds \(per\-seed stability is small; see the released logs\), andcsrand recovery, our main axes, are unaffected by generation temperature in the sense that trapped calls are detected structurally from the call trace\.

## 6Results

### 6\.1Susceptibility scales with capability

Table[1](https://arxiv.org/html/2608.04719#S6.T1)reports the declared\-condition results\. Per\-taskcsrranges from0\.0100\.010\(Opus 4\.8\) to0\.3780\.378\(Llama 3\.1 8B\), a36×36\\timesspread, with non\-overlapping bootstrap confidence intervals separating the strongest models from the weakest\. Aχ2\\chi^\{2\}test confirmscsrdiffers across models \(χ2=1467\.0\\chi^\{2\}\\\!=\\\!1467\.0,dof=7\\mathrm\{dof\}\\\!=\\\!7,p<0\.001p\\\!<\\\!0\.001\); because theχ2\\chi^\{2\}pools individual calls, we also run a task\-clustered Kruskal\-Wallis test on per\-taskcsr, which treats each task\-run rather than each call as the unit and agrees \(H=481\.9H\\\!=\\\!481\.9,p<0\.001p\\\!<\\\!0\.001; Appendix[E](https://arxiv.org/html/2608.04719#A5)\)\. The weakest model \(Llama 3\.1 8B\) fails most canaries, barely completes tasks \(tsr=0\.23=0\.23\), and rarely recovers \(0\.180\.18\)\.

Table 1:Declared\-condition results, sorted bycsr\.csris the mean per\-task canary susceptibility rate with bootstrap 95% CIs;tsris task success as graded by the provider\-independent judge; Rec\. is recovery rate with its denominatornn\(the number of trapped tasks\)\. Lowercsrand highertsr/Rec\. are better\.csrand recovery are judge\-independent; onlytsrdepends on the judge\. Recovery for the most robust models rests on few trapped tasks \(n≤41n\\\!\\leq\\\!41\) and is indicative\.Susceptibility is driven far more by model identity than by task difficulty \(Figure[4](https://arxiv.org/html/2608.04719#S6.F4)\)\. Most susceptible models peak on medium\-difficulty tasks, and Llama 3\.1 8B actually traps*less*on hard tasks because it makes fewer tool calls there; Qwen3\-8B is the exception, spiking on hard tasks as it spirals into repeated calls\. The resistant models stay low across all three levels\.

![Refer to caption](https://arxiv.org/html/2608.04719v1/figures/ctr_by_difficulty.png)Figure 4:Per\-taskcsrby task difficulty \(declared\)\. Model identity, not difficulty, dominates: most susceptible models peak on medium tasks, and Llama 3\.1 8B traps less on hard tasks \(where it makes fewer calls\), while Qwen3\-8B spikes there\.
### 6\.2Capability tier does not predict safety

Capability tier is a poor predictor of canary resistance \(Figure[2](https://arxiv.org/html/2608.04719#S1.F2)\)\. GPT\-4\.1 \(mid\) is the most susceptible of the six hosted models, exceeding all three frontier models including its own provider’s GPT\-5\.2\. Within Google, the mid\-tier Gemini 2\.5 Flash \(0\.0410\.041\)*outperforms*the frontier Gemini 2\.5 Pro \(0\.0490\.049\)\. The two Anthropic models \(Opus, Sonnet\) are the most resistant regardless of tier, suggesting provider and training matter more than nominal capability tier\. We state this as an observation over eight models with vendor\-assigned tier labels, not a statistical law: withn=8n\\\!=\\\!8the point is that nominal tier fails to order canary resistance, not that capability is irrelevant \(within a provider, the stronger model is not reliably safer\)\.

### 6\.3A capability\-stratified taxonomy

The per\-type trap rates \(Figure[5](https://arxiv.org/html/2608.04719#S6.F5), Table[2](https://arxiv.org/html/2608.04719#S6.T2)\) reveal why a six\-type taxonomy is warranted even though most types look inert on strong models; the per\-model profiles in Figure[6](https://arxiv.org/html/2608.04719#S6.F6)make the same point at a glance, collapsing to a single capability\-mirage spoke for hosted models and filling out for the small ones\. Averaged over the six hosted models,*capability mirage*is the only strongly active probe \(0\.2160\.216\);*prerequisite blindness*is a distant second \(0\.0940\.094\) and the remaining four types are near\-inert \(0\.0080\.008–0\.0270\.027\)\. Yet on the small open models nearly every type fires \(Llama 3\.1 8B0\.1060\.106–0\.2550\.255; Qwen3\-8B0\.0240\.024–0\.2920\.292\), and the types that look inert on hosted models are precisely the ones that jump most \(on Llama, temporal0\.009→0\.2550\.009\\\!\\rightarrow\\\!0\.255; parameter0\.008→0\.2450\.008\\\!\\rightarrow\\\!0\.245\)\. The probes are therefore not weak; they discriminate by capability\. Holm\-corrected Fisher exact tests find 64/168 pairwise type×\\timesmodel contrasts significant atα=0\.05\\alpha\\\!=\\\!0\.05\. Appendix[F](https://arxiv.org/html/2608.04719#A6)gives one representative trapped run per type, showing the distinct diagnosis each yields\.

Table 2:Type\-specific trap rate \(declared\): mean over the six hosted models vs\. the two small open models\. Capability mirage catches strong models; the other probes are near\-inert on hosted models but fire on the small models\.![Refer to caption](https://arxiv.org/html/2608.04719v1/figures/type_ctr_heatmap.png)Figure 5:Type\-specific trap rate, models×\\timescanary types \(declared\)\. The diagnostic centerpiece: capability mirage is the only column active across strong models; the small model lights up the full taxonomy\.![Refer to caption](https://arxiv.org/html/2608.04719v1/figures/radar_profiles.png)Figure 6:Per\-model diagnostic profiles \(small multiples; shared radial axis\)\. Hosted models reduce to a single capability\-mirage spoke; the small open models show full six\-spoke profiles\.
### 6\.4Canaries predict failure; robust models resist

Per\-taskcsrcorrelates negatively with task success \(Spearmanρ=−0\.34\\rho\\\!=\\\!\-0\.34,p<0\.001p\\\!<\\\!0\.001,n=2880n\\\!=\\\!2880\), validating canary susceptibility as a diagnostic for real degradation\. Because tool outputs are synthetic,tsrmeasures whether the agent completes the task*procedurally*rather than whether the final fact is correct, so this link is between canary susceptibility and procedural failure specifically\. Since the 120 tasks recur across models and seeds, we recompute the correlation with a task\-clustered bootstrap \(resampling whole tasks, 120 clusters\): the interval still excludes zero \(ρ=−0\.34\\rho\\\!=\\\!\-0\.34\[−0\.40,−0\.28\-0\.40,\-0\.28\]\), so the association is not an artifact of task reuse\. We test the task\-success delta under canary pressure \(baseline minus condition; Figure[7](https://arxiv.org/html/2608.04719#S6.F7)\) with a bootstrap 95% CI per model\. The degradation is significant and positive \(canaries hurt\) for the OpenAI, Google, and small models\. GPT\-5\.2 \(declared\), for example, has Cohen’sd=\+0\.23d\\\!=\\\!\+0\.23withΔ=\+0\.11\\Delta\\\!=\\\!\+0\.11\[\+0\.04,\+0\.17\+0\.04,\+0\.17\], and Llama hasΔ=\+0\.10\\Delta\\\!=\\\!\+0\.10\[\+0\.04,\+0\.17\+0\.04,\+0\.17\]\. Under the full condition the two small models degrade most sharply \(Qwen3\-8BΔ=\+0\.48\\Delta\\\!=\\\!\+0\.48\[\+0\.42,\+0\.54\+0\.42,\+0\.54\],d=\+1\.17d\\\!=\\\!\+1\.17; LlamaΔ=\+0\.29\\Delta\\\!=\\\!\+0\.29\[\+0\.24,\+0\.34\+0\.24,\+0\.34\]\)\. For the two Anthropic models the interval*includes zero*\(OpusΔ=−0\.02\\Delta\\\!=\\\!\-0\.02\[−0\.08,\+0\.04\-0\.08,\+0\.04\]; SonnetΔ=0\.00\\Delta\\\!=\\\!0\.00\[−0\.06,\+0\.05\-0\.06,\+0\.05\]\), so the most robust models show no significant degradation under canary pressure\. Since the effect is not significant, we read it as robustness to the added tools rather than as canaries somehow improving performance\.

![Refer to caption](https://arxiv.org/html/2608.04719v1/figures/tsr_delta.png)Figure 7:Task\-success degradation \(baseline minus canary condition\)\. Positive means canaries hurt\. The OpenAI, Google, and small models degrade significantly; the Anthropic models’Δ\\Deltais not significantly different from zero \(bootstrap CI includes 0\)\.
### 6\.5Recovery

Recovery after a trap \(Table[1](https://arxiv.org/html/2608.04719#S6.T1)\) scales with capability: Opus self\-corrects 82% of the time, Llama 3\.1 8B only 18%\. Recovery is thus a second, capability\-aligned diagnostic axis distinct from raw susceptibility\. The robust models trip few traps, so their recovery rests on small denominators \(n≤41n\\\!\\leq\\\!41; Table[1](https://arxiv.org/html/2608.04719#S6.T1)\) and is best read as indicative\. Recovery clearly pays off: a trapped run that recovers succeeds about half the time, against roughly one in six when it does not \(Appendix[G](https://arxiv.org/html/2608.04719#A7)\)\.

### 6\.6Effect of canary density

Table[3](https://arxiv.org/html/2608.04719#S6.T3)reports thefullcondition, where all six canary types are injected for every task\. Counter\-intuitively, for the hosted modelscsris no higher than indeclared, and usually lower \(e\.g\. GPT\-5\.20\.178→0\.1180\.178\\\!\\rightarrow\\\!0\.118\)\. This is not a denominator artifact: the per\-type rate of capability mirage, the dominant trap, drops declared\-to\-full for every model \(GPT\-4\.10\.579→0\.2860\.579\\\!\\rightarrow\\\!0\.286\)\. With more competing canary types present, capable models appear to read tool descriptions more carefully, so raw canary*density*is not the same as diagnostic*pressure*\. The small open models behave oppositely under saturation: Qwen3\-8B’scsrmore than doubles \(0\.291→0\.7150\.291\\\!\\rightarrow\\\!0\.715\) as it spirals into repeated canary calls, while Llama collapses in the other direction, making so few calls it barely completes any task \(tsr0\.040\.04\)\. We therefore report declared as the primary condition, since it concentrates each task’s relevant probes\.

Table 3:Full\-condition results \(all six canary types injected per task\), same columns as Table[1](https://arxiv.org/html/2608.04719#S6.T1)\. For hosted modelscsris no higher than under declared, as extra types dilute the dominant capability\-mirage probe; the small open models instead saturate \(Qwen3\-8B spikes, Llama collapses to near\-zero tool use\)\.
### 6\.7Probes survive de\-telling

A natural worry is that a strong model’s lowcsrreflects spotting an obvious give\-away phrase \(every semantic decoy states it “returns cached data,” every capability mirage claims “research\-grade” powers\) rather than genuine reasoning\. We test this with a controlled ablation: a second canary pool*identical*in names, ids, schemas, tasks, seeds, and judge, changing*only*the description phrasing of the three phrase\-based types \(softening the tell\), and re\-run the declared condition for all eight models \(2,8802\{,\}880runs\)\. If the low frontier rates were mere phrase\-spotting, removing the phrase should raise them\. It does not: frontiercsris essentially unchanged \(0\.079→0\.0750\.079\\\!\\rightarrow\\\!0\.075; Figure[8](https://arxiv.org/html/2608.04719#S6.F8)\), so the near\-zero frontier rates reflect tool\-selection reasoning, not tell\-detection\.csrin fact falls slightly for most models, concentrated in capability mirage \(e\.g\. GPT\-4\.10\.579→0\.4920\.579\\\!\\rightarrow\\\!0\.492\), because its boastful wording is*both*the tell and the lure; softening it makes the mirage less tempting as well as less obvious\. The ablation therefore does not fully separate tell from lure, but the key conclusion is robust: with a plausible, non\-boastful mirage, frontiercsrstays near\-zero\.

![Refer to caption](https://arxiv.org/html/2608.04719v1/figures/subtle_ablation.png)Figure 8:Subtlety ablation \(declared\): per\-taskcsron standard vs\. softened \(subtle\) canaries\. Softening the give\-away phrase does not raise the trap rate of the strong models \(frontier average0\.079→0\.0750\.079\\\!\\rightarrow\\\!0\.075\), so their low rates are not an artifact of phrase\-spotting\.

## 7Discussion

The clearest practical lesson is that*a model’s capability tier is not a safety guarantee for tool selection*\. Trading up to a bigger model in the same family does not reliably buy safer tool choice, and can even hurt: within OpenAI the frontier GPT\-5\.2 is the safer of the pair, yet within Google it is the cheaper Gemini Flash that resists best, and the single most canary\-prone hosted model, GPT\-4\.1, sits in the mid tier\. Canary injection gives teams a cheap pre\-deployment readiness check, since a high type\-specific trap rate points to*which*reasoning weakness to harden\. High capability\-mirage susceptibility, for instance, argues for guardrails on “more\-powerful” tool variants\. Capability mirage is also the most persistent trap: it reaches further up the capability curve than any other probe, still catching mid\-tier and weaker frontier models, with only prerequisite blindness rivalling it and only on GPT\-5\.2\. It fades only at the very top \(Opus barely trips it, at a rate that edges lower still once the boastful wording is softened; §[6\.7](https://arxiv.org/html/2608.04719#S6.SS7)\)\. This suggests that inflated capability claims in tool descriptions are a systemic vulnerability the MCP ecosystem should standardize against, and that even strong models are not uniformly immune\.

Beyond diagnosis, the results point to a few concrete levers\. Layering a small panel of typed canaries onto an existing tool\-use suite upgrades its binary pass/fail into a reason code at almost no cost, which is an easy win for benchmark builders\. On the architecture side, recovery being a separate, capability\-aligned axis makes the case for an explicit verify\-and\-backtrack step: a model that springs a canary but recovers loses little, so even a cheap post\-call check, such as noticing a staleness or reliability flag in a result, can turn a trap into a near\-miss\. And for anyone deploying agents, the capability\-mirage finding reads as a simple description\-hygiene rule, namely to avoid the superlative capability claims in tool descriptions that even frontier models over\-trust\. More broadly, we would caution against reading tool\-selection safety off a model’s general tier; each deployment is better probed directly\.

## 8Conclusion

Canary tools turn a binary “wrong tool” outcome into a capability\-stratified diagnostic profile\. Across eight models we find that susceptibility scales steeply with capability, that tier does not predict safety, and that a six\-type taxonomy earns its keep because each probe discriminates at a different point on the capability curve, with capability mirage reaching furthest up it\. We release the framework so that others can run canary\-based readiness checks on their own agent deployments\.

## Limitations

Tool outputs are realistic but synthetic; tasks requiring a specific real\-world fact are scored against the agent’s ability to proceed coherently, not factual correctness, which uniformly lowerstsron such tasks across all models\. The small tier is two open\-weight 8B models \(Llama 3\.1 8B and Qwen3\-8B\) at three seeds each; broader coverage of small and mid\-size open models would further strengthen the small\-tier claims\. The suite is 120 single\-authored, templated tasks, which is modest for a benchmark; we intend it as a seed suite for an extensible framework \(the generator produces canaries for any tool set\) rather than a comprehensive benchmark, and our statistics rest on 8,640 task\-runs with hundreds of per\-type trap opportunities rather than on 120 raw points\. The six hosted models come from three providers \(one frontier \+ one mid each\), so provider and tier are partially confounded\.tsris graded by a provider\-independent judge, corroborated by an independent second judge \(GLM\-5,κ=0\.75\\kappa\\\!=\\\!0\.75\) and by a 40\-run hand check \(κ=0\.90\\kappa\\\!=\\\!0\.90, all disagreements in the conservative direction; §[4](https://arxiv.org/html/2608.04719#S4)\); a larger, fully blind human study would strengthen this further\. The subtlety ablation \(§[6\.7](https://arxiv.org/html/2608.04719#S6.SS7)\) softens the give\-away phrase but does not fully separate a canary’s tell from its lure, since for capability mirages the two coincide\.

## References

- Anthropic \(2024\)Anthropic\. 2024\.Introducing the model context protocol\.[https://www\.anthropic\.com/news/model\-context\-protocol](https://www.anthropic.com/news/model-context-protocol)\.
- Bandi et al\. \(2026\)Chaithanya Bandi, Razvan\-Gabriel Dumitru, Ben Hertzberg, Divyansh Agarwal, et al\. 2026\.MCP\-Atlas: A large\-scale benchmark for tool\-use competency with real MCP servers\.*arXiv preprint arXiv:2602\.00933*\.
- Huang et al\. \(2024\)Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun\. 2024\.MetaTool benchmark for large language models: Deciding whether to use tools and which to use\.In*International Conference on Learning Representations \(ICLR\)*\.
- Liu et al\. \(2025\)Wenrui Liu, Zixiang Liu, Elsie Dai, Wenhan Yu, Lei Yu, Tong Yang, Jinjun Han, and Hong Gao\. 2025\.MCPAgentBench: A real\-world task benchmark for evaluating LLM agent MCP tool use\.*arXiv preprint arXiv:2512\.24565*\.
- Liu et al\. \(2024\)Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, et al\. 2024\.AgentBench: Evaluating LLMs as agents\.In*International Conference on Learning Representations \(ICLR\)*\.
- Lu et al\. \(2024\)Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang\. 2024\.ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities\.*arXiv preprint arXiv:2408\.04682*\.
- Patil et al\. \(2025\)Shishir G\. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng\-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E\. Gonzalez\. 2025\.The Berkeley function calling leaderboard \(BFCL\): From tool use to agentic evaluation of large language models\.In*Proceedings of the 42nd International Conference on Machine Learning \(ICML\)*\.
- Patil et al\. \(2024\)Shishir G\. Patil, Tianjun Zhang, Xin Wang, and Joseph E\. Gonzalez\. 2024\.Gorilla: Large language model connected with massive APIs\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Qin et al\. \(2024\)Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, et al\. 2024\.ToolLLM: Facilitating large language models to master 16000\+ real\-world APIs\.In*International Conference on Learning Representations \(ICLR\)*\.
- Schick et al\. \(2023\)Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\. 2023\.Toolformer: Language models can teach themselves to use tools\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Yao et al\. \(2024\)Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan\. 2024\.τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains\.*arXiv preprint arXiv:2406\.12045*\.
- Yao et al\. \(2023\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao\. 2023\.ReAct: Synergizing reasoning and acting in language models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Zhan et al\. \(2024\)Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang\. 2024\.InjecAgent: Benchmarking indirect prompt injections in tool\-integrated large language model agents\.In*Findings of the Association for Computational Linguistics: ACL 2024*\.
- Zhang et al\. \(2024\)Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, and Hayato Yamana\. 2024\.ToolBeHonest: A multi\-level hallucination diagnostic benchmark for tool\-augmented large language models\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*\.

## Appendix AModels and versions

Table[4](https://arxiv.org/html/2608.04719#A1.T4)lists the exact model identifiers behind the short names used in the paper, for reproducibility\. The six hosted models are served through an OpenAI\-compatible gateway; the two small open\-weight models run locally via Ollama\. All models are queried through one shared tool\-calling loop with each provider’s default decoding settings, and the three seeds permute tool ordering\. Task success is graded by DeepSeek\-V3\.2 at temperature 0, with GLM\-5 as the independent cross\-judge; neither is one of the eight tested models, and both are listed with their identifiers at the foot of Table[4](https://arxiv.org/html/2608.04719#A1.T4)\.

ModelProvider \(tier\)API identifierOpus 4\.8Anthropic \(frontier\)anthropic/claude\-opus\-4\-8GPT\-5\.2OpenAI \(frontier\)openai/gpt\-5\.2Gemini 2\.5 ProGoogle \(frontier\)vertex\_ai/gemini\-2\.5\-proSonnet 4\.6Anthropic \(mid\)anthropic/claude\-sonnet\-4\-6GPT\-4\.1OpenAI \(mid\)openai/gpt\-4\.1Gemini 2\.5 FlashGoogle \(mid\)vertex\_ai/gemini\-2\.5\-flashQwen3\-8BAlibaba \(small\)qwen3:8b\(Ollama\)Llama 3\.1 8BMeta \(small\)llama3\.1:8b\(Ollama\)Judges \(not among the evaluated models\)DeepSeek\-V3\.2DeepSeek \(primary\)vertex\_ai/deepseek\-ai/deepseek\-v3\.2\-maasGLM\-5Z\.ai \(cross\-judge\)vertex\_ai/zai\-org/glm\-5\-maas

Table 4:The eight evaluated models with their exact API identifiers, and the two judge models \(primary and independent cross\-judge\) used to grade task success\. Short names in the paper map to these identifiers\.
## Appendix BOutcome\-judge prompt

The judge receives the task, the canonical tool hint, the full tool\-call trace, and the final answer, and returns a JSON verdict\. We grade at temperature 0 for reproducibility \(§[4](https://arxiv.org/html/2608.04719#S4)\)\. The verbatim template is:

> You are grading whether an AI agent accomplished a task\. TASK GIVEN TO THE AGENT: \{prompt\} TOOLS THAT WOULD NORMALLY ACCOMPLISH IT \(hint, not a requirement; the agent may solve a step by reasoning or use an equivalent tool\): \{correct\_tools\} THE AGENT’S TOOL\-CALL TRACE \(name, arguments, result\): \{trace\} THE AGENT’S FINAL ANSWER: \{final\} Did the agent accomplish what the task asked? Judge the OUTCOME, not which tools it picked\. It succeeds if the final answer and trace deliver what the user asked for, even via a different path\. It fails if it gave up, stalled, produced a wrong/empty result, or only described what it would do\. Respond with ONLY a JSON object: \{"success": true\|false, "reason": "\.\.\."\}

## Appendix CCanary generation

Each real tool yields one canary per type\. Four types are deterministic schema transforms:*parameter*renames arguments and injects an unsatisfiable requiredapi\_key;*prerequisite*strips authentication language and marks the tool as spanning private resources;*temporal*appends an outdated date/version;*granularity*removes parameters and hardcodes a special case\. Two types \(*semantic*,*capability*\) additionally use an LLM to reword the name and description so they are not trivially distinguishable\. The give\-away suffix each phrase\-based type appends is fixed; the subtlety ablation \(§[6\.7](https://arxiv.org/html/2608.04719#S6.SS7)\) swaps only these suffixes, e\.g\. capability mirage “ADVANCED: solves the hardest cases with research\-grade accuracy” becomes “optimized variant with extended precision\.” The full pool \(72 canaries\) is generated once and released\.

## Appendix DHuman validation of the judge

An author graded a stratified sample of 40 task\-runs \(spanning models and all difficulties\), reading the prompt, the full trace, and the final answer\. Human verdicts agree with the judge on 95% \(Cohen’sκ=0\.90\\kappa\\\!=\\\!0\.90\)\. Both disagreements are cases where the judge was*stricter*than the human: \(i\) two identical Opus runs received opposite judge verdicts, which motivated our temperature\-0 judging and is now resolved \(under temperature 0 both runs receive the same verdict\); \(ii\) a task whose tax was computed on a stated $5000 profit was failed for not using an unrelated Bitcoin\-price lookup\. No case had the judge crediting a run the human would fail, sotsris conservative\. The full sheet is released with the framework\.

## Appendix ECluster\-robust tests and judge stability

Because tool calls within a run and the 120 tasks across models/seeds are not independent, we complement the call\-level tests with cluster\-robust ones: a Kruskal\-Wallis test on*per\-task*csracross models \(H=481\.9H\\\!=\\\!481\.9,p<0\.001p\\\!<\\\!0\.001, declared\) and a*task\-clustered*bootstrap for thecsr–tsrcorrelation that resamples whole tasks \(ρ=−0\.34\\rho\\\!=\\\!\-0\.34, 95% CI\[−0\.40,−0\.28\]\[\-0\.40,\-0\.28\]over 120 clusters\)\. For judge stability, we re\-grade a 300\-run sample three times at temperature 0\.7: the three samples are unanimous on all sampled runs and agree with the deterministic temperature\-0 verdict on 94%, so the primary verdict is stable\. The independent cross\-judge \(GLM\-5\) agreement is also stable across canary types: per\-type Cohen’sκ\\kapparanges from0\.730\.73\(parameter trap\) to0\.790\.79\(temporal decoy\), so no single probe type drives the pooled agreement \(κ=0\.75\\kappa\\\!=\\\!0\.75\)\. Per\-run votes are released alongside the logs\.

## Appendix FRepresentative failures

Table[5](https://arxiv.org/html/2608.04719#A6.T5)shows one real trapped run per canary type, drawn from frontier to 8B models\. Each illustrates the typed diagnosis a binary distractor score would collapse to “wrong tool\.”

Table 5:One representative trapped run per canary type\. Each wrong pick localizes a distinct reasoning weakness that a binary “wrong tool” score would hide\.
## Appendix GWhy recovery matters

Recovering from a trap is what saves the task\. Across all 1,313 trapped runs \(pooled over models, declared and full\), a run in which the agent caught itself and switched to a correct tool succeeded on the task about half the time, whereas a run that stayed on the canary succeeded only about one time in six \(Figure[9](https://arxiv.org/html/2608.04719#A7.F9)\)\. Recovery roughly triples the odds of still finishing the task, which is why we treat it as a second diagnostic axis alongside raw susceptibility: two models with the same trap rate are not equally safe if one reliably backs out and the other does not\.

![Refer to caption](https://arxiv.org/html/2608.04719v1/figures/recovery_outcome.png)Figure 9:Task outcome for trapped runs, split by whether the agent recovered\. Recovering from a trap lifts task success from 16% to 52%\.