TRACE: Business Rule-Grounded Reasoning Curriculum for Knowledge-Preserving Parametric Tool Retrieval in Enterprise LLMs

Hugging Face Daily Papers Papers
parametric-retrieval tool-retrieval enterprise-llms chain-of-thought reasoning-curriculum knowledge-preservation lora

Summary

TRACE introduces a two-stage curriculum to preserve parametric tool knowledge in enterprise LLMs while enabling fast single-beam greedy decoding, achieving improved accuracy and recall over baselines.

Parametric retrieval enables LLMs to retrieve tools implicitly by assigning each API a unique virtual token and training the model to generate it via constrained beam search. Toolsense shows that this regime has two critical drawbacks: it destroys parametric tool knowledge during training, and its beam-search decoding is too slow for real-time deployment. We introduce TRACE (Tool Retrieval via Augmented Chain-of-thought and Enterprise rules), a two-stage curriculum that resolves this dissociation. Stage 1 reuses the multi-format memorization SFT from ToolSense to seed tool knowledge with LoRA. Stage 2 is our core contribution: the model is trained to emit a thinking trace before producing a JSON list of tool tokens, using two data sources -- RRB pairs from ToolSense and queries synthesized to target business rules curated by domain experts -- both augmented with reasoning traces. This training objective preserves Stage 1 MCQ and QA probing accuracy while enabling single-beam greedy decoding at production latency. Evaluated on a combined enterprise catalog of 8,300+ tools across two enterprise product lines, TRACE training for Stage 2 not only preserves but improves tool understanding: MCQ accuracy gains +3.2 pp and QA probing gains +9 pp over Stage 1. On retrieval, TRACE achieves ~86% recall on Domain A and ~60% on Domain B -- compared to embedding baseline performance of ~27% & ~52% -- both with single-beam greedy decoding, making it directly deployable at production latency.
Original Article
View Cached Full Text

Cached at: 07/28/26, 02:25 PM

Paper page - TRACE: Business Rule-Grounded Reasoning Curriculum for Knowledge-Preserving Parametric Tool Retrieval in Enterprise LLMs

Source: https://huggingface.co/papers/2607.22639

Abstract

ParametricretrievalenablesLLMstoretrievetoolsimplicitlybyassigningeachAPIauniquevirtualtokenandtrainingthemodeltogenerateitviaconstrainedbeamsearch.Toolsenseshowsthatthisregimehastwocriticaldrawbacks:itdestroysparametrictoolknowledgeduringtraining,anditsbeam-searchdecodingistooslowforreal-timedeployment.WeintroduceTRACE(ToolRetrievalviaAugmentedChain-of-thoughtandEnterpriserules),atwo-stagecurriculumthatresolvesthisdissociation.Stage1reusesthemulti-formatmemorizationSFTfromToolSensetoseedtoolknowledgewithLoRA.Stage2isourcorecontribution:themodelistrainedtoemitathinkingtracebeforeproducingaJSONlistoftooltokens,usingtwodatasources--RRBpairsfromToolSenseandqueriessynthesizedtotargetbusinessrulescuratedbydomainexperts--bothaugmentedwithreasoningtraces.ThistrainingobjectivepreservesStage1MCQandQAprobingaccuracywhileenablingsingle-beamgreedydecodingatproductionlatency.Evaluatedonacombinedenterprisecatalogof8,300+toolsacrosstwoenterpriseproductlines,TRACEtrainingforStage2notonlypreservesbutimprovestoolunderstanding:MCQaccuracygains+3.2ppandQAprobinggains+9ppoverStage1.Onretrieval,TRACEachieves~86%recallonDomainAand~60%onDomainB--comparedtoembeddingbaselineperformanceof~27%&~52%--bothwithsingle-beamgreedydecoding,makingitdirectlydeployableatproductionlatency.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.22639

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.22639 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.22639 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.22639 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs

arXiv cs.CL

This paper investigates whether reinforcement learning can improve the direct recall of parametric knowledge in LLMs beyond reasoning tasks. It demonstrates that RL with binary rewards yields significant gains in factual QA benchmarks by redistributing probability mass to unlock latent knowledge rather than acquiring new facts.

Stepwise Reasoning Enhancement for LLMs via External Subgraph Generation

arXiv cs.CL

This paper proposes SGR, a framework that enhances LLM stepwise reasoning by integrating external knowledge graphs through query-relevant subgraph generation, combining Cypher-based reasoning with collaborative reasoning integration. Experiments on CWQ, WebQSP, GrailQA, and KQA Pro show improved reasoning accuracy over standard prompting and knowledge-enhanced baselines.