Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
Summary
The paper shows that keyword-matching benchmarks can credit small language models with tool use they never perform, documenting a false positive in matched-architecture Spanish security LMs and proposing a cheap diagnostic ladder plus a targeted SFT recipe that repairs the failing 1B model.
View Cached Full Text
Cached at: 10/02/26, 08:31 PM
Paper page - Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
Source: https://huggingface.co/papers/2610.02142
Abstract
Keyword-matchingbenchmarkscancreditsmallmodelsfortoolusetheyneverperform.Wedocumentsuchafalsepositiveinamatched-architecturepairofSpanishsecuritylanguagemodelsandproposealadderofstrict,cheapdiagnostics.A661.6Mparametermodel(approx.65%code/technicaltext;nodedicatedSFT)anda1,109Mmodel(web-heavymulti-phasecurriculum;6B-tokentool-SFT)sharedecoder,tokenizer,andspecialtokens,scoringalmostidenticallyonlenienttool-usemetrics(B4:0.660vs.0.650).Verbatim-reproductionchecksontrainingexamplesseparatethemcompletely:the600Memitsvalidtoolcallswithgeneralizedargumentson6/6examples;the1Bdoessoon0/6acrosscheckpoints.Afirst-tokenprobelocalizesthe1B’sfailuretoamissingprior(prob.10^{-4}--10^{-5}on<|tool_call|>),whichwaserasedbyitsweb-heavytrainingphase.AtargetedSFTrecipe(diversecorpus,5xhigherlearningrate,2,202steps,~3.3GPU-hours)repairsthe1Busingthreeordersofmagnitudefewertokensthanthefailedphase.Onall269corpusrows,validemissionrisesfrom0.100to0.959(600M:0.926).On238unseenprompts,therepaired1Bpasses0.536vs.the600M’s0.428(p=0.004).Embedding-driftchecksshowtherepairdidnotmovethetriggertoken’stiedembedding(97.7%ofthebf16tableremainsbit-identical),meaningchangesliveinthesurroundingnetwork.Bothmodelsover-trigger,rarelyansweringnegativepromptswithoutacall(0.09for600M,0.17forrepaired1B).Factorialanalysesconfirmallrepairconfigurationsinstalltheformat,thoughsuppressionbenefitsfromadiversecorpusremainahypothesisduetoseedsensitivity.ThischeapdiagnosticladdercostsminutesofCPUtimeandshouldgatetool-useclaimsonsmallmodels.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2610\.02142
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.02142 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2610.02142 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.02142 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models
This paper evaluates automated safety benchmarks for small language models, finding high ambiguity in judgments that compromises reliability and reveals a capability-safety confound.
Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
This paper synthesizes 27 benchmark, taxonomy, and audit papers from 2023-2026 into a unified taxonomy of LLM agent limitations, identifying six failure clusters including tool invocation errors, planning failures, long-horizon degradation, multi-agent coordination issues, safety concerns, and measurement validity problems.
Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models
This paper proposes using lightweight MLP probes on activations from LLaMA-3.1-8B to detect harmful prompts with high F1 scores, offering a cost-effective alternative to large external guardrail models.
Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy
This paper introduces AI-MASLD, a stress-audit framework for medical LLMs that reveals how benchmark accuracy can hide serious safety failures, and demonstrates that open-weight models can match or exceed proprietary ones on safety dimensions.
Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use
This paper introduces PhysTool-Bench, a benchmark for evaluating multimodal large language models' ability to recognize and plan the use of physical tools in real-world scenes. The authors find that even the best model identifies only 58.7% of tools and completes just 21.0% of queries end-to-end, revealing a two-level deficit in perception and functional commonsense.