OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling

# OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling
Source: [https://arxiv.org/html/2608.00814](https://arxiv.org/html/2608.00814)
###### Abstract

LLMs generate tool calls token by token, even though the function choice and argument values can often be predicted in parallel from the request and tool schema\.ToolSpecreduces this cost by drafting schema tokens and retrieving earlier calls, but cannot propose request\-specific values absent from either source\. We presentOoO\-Spec, which computes these missing semantics out of order\. At request arrival, a Qwen3\-0\.6B sidecar predicts the function choice and all schema\-defined argument slots in one parallel request\-level wave while the target beginsToolSpecdecoding\. The runtime joins the slot values, renders the resulting call as text, and exposes it to subsequent candidate\-construction rounds\. The target polls without blocking, re\-tokenizes a ready hint with its own tokenizer, and remains the sole verifier and commit authority\. The sidecar is trained once with LoRA on Qwen2\.5\-32B teacher traces and used unchanged across Qwen2\.5, Qwen3, and Llama targets, without target\-specific drafter training\. Across seven fully ranked targets and three benchmarks under greedy batch\-one decoding,OoO\-Specis fastest among all evaluated methods in all 21 target–benchmark cells, reaching2\.46×2\.46\\times–5\.34×5\.34\\timesover autoregressive decoding with an unweighted mean of3\.89×3\.89\\times, versus2\.95×2\.95\\timesforToolSpec\. It also outperforms every evaluated released learned drafter in each comparable cell\. Across Qwen3\-4B, 8B, 14B, and 32B targets, the same sidecar improves onToolSpecby 34\.1% on average\. Its compact semantic payload averages 85 bytes per request excluding protocol metadata, supporting effective split\-GPU overlap\.

## 1Introduction

Large language models are frequently used fortool calling, repeatedly generating strict, often lengthy function calls\. Although each call is a structured object, the decoder treats it as a flat token sequence\. Autoregressive decoding therefore requires a forward pass of the target model for every token in its function name, field names, delimiters, and argument values\. Yet the function and many argument values can often be inferred directly from the request and schema, well before the decoder reaches their positions in the output\. Thus, the target generates the call sequentially even though much of its content can be computed earlier or in parallel\.

![Refer to caption](https://arxiv.org/html/2608.00814v1/x1.png)Figure 1:Autoregressive drafting versus out\-of\-order speculation\. Top: an autoregressive drafter proposes tokens one at a time, while an out\-of\-order drafter resolves independent slots in parallel\. Bottom: the sidecar runs alongside the target timeline and sends its result as a hint that joins speculative decoding as additional draft candidates\.Speculative decodingreduces this serial cost by proposing multiple future tokens and verifying them together with the target\(Leviathanet al\.[2023](https://arxiv.org/html/2608.00814#bib.bib1); Chenet al\.[2023](https://arxiv.org/html/2608.00814#bib.bib2)\)\. Learned drafters such as EAGLE\-3, PARD\-2, and DFlash can produce strong proposals, but their checkpoints or proposal interfaces are closely tied to a target model\(Liet al\.[2025](https://arxiv.org/html/2608.00814#bib.bib5); Anet al\.[2026](https://arxiv.org/html/2608.00814#bib.bib10); Chenet al\.[2026](https://arxiv.org/html/2608.00814#bib.bib11)\)\. Other methods avoid training a target\-specific drafter by reusing tokens from the prompt, generation history, or schema\(Saxena[2023](https://arxiv.org/html/2608.00814#bib.bib8); Luoet al\.[2025](https://arxiv.org/html/2608.00814#bib.bib6); Huet al\.[2024](https://arxiv.org/html/2608.00814#bib.bib7)\)\.ToolSpecbuilds specifically on two properties of tool calling: most output tokens are fixed by the schema, and similar calls recur across requests\(Xiaet al\.[2026](https://arxiv.org/html/2608.00814#bib.bib9)\)\. It uses a finite\-state machine \(FSM\) to alternate between schema\-token filling and speculative generation for variable fields, and retrieves similar completed calls as additional draft candidates\. The target packs these candidates into a tree and verifies them over repeated decoding rounds\.

However,ToolSpeccan only reuse what already exists: schema tokens and values retrieved from History Calls\. Values new to the current request still fall back to autoregressive decoding\. Addressing this gap raises three challenges\.\(1\) Generate unseen values\.Schema filling provides syntax, while retrieval only reuses earlier calls\. A proposer must infer new request\-specific values\.\(2\) Generate in parallel\.As we observed, semantic fields need not be computed in textual order\. A proposer must run in parallel without blocking the target, so that its latency does not enter the decoding path\.\(3\) Use whenever ready\.A concurrent proposal becomes ready at an unpredictable point in the target’s decoding, and the target never waits for it\. To make full use of concurrency, the system must be able to use a proposal no matter when it becomes ready\.

Based on these observations and challenges, we introduceOoO\-Spec, an asynchronous semantic speculation system for tool calls\. As Figure[1](https://arxiv.org/html/2608.00814#S1.F1)shows, a lightweight sidecar resolves the function and argument values as independent slots, in parallel and out of textual order, while the target immediately begins the decoding loop\. Once ready, the sidecar’s outputs join a later candidate\-construction boundary as a current\-request semantic hint\. The target never waits for the hint and remains responsible for verifying and committing every token\.

Our sidecar is Qwen3\-0\.6B\(Yanget al\.[2025](https://arxiv.org/html/2608.00814#bib.bib18)\)with one LoRA adapter trained on Qwen2\.5\-32B teacher traces\(Qwen Team[2024](https://arxiv.org/html/2608.00814#bib.bib17); Huet al\.[2022](https://arxiv.org/html/2608.00814#bib.bib16)\)\. We reuse this sidecar—base draft model and adapter together—unchanged across all targets and workloads\. This portability applies to the learned proposer; each target keeps its ownToolSpecintegration, tokenizer, and frozen rendering policy\. The sidecar returns only request\-level semantic strings, so it can serve every target and run asynchronously on a separate device\.

We evaluate seven targets from the Qwen2\.5, Qwen3, and Llama families on API\-Bank, ToolAlpaca, and BFCL under greedy batch\-one decoding\(Liet al\.[2023](https://arxiv.org/html/2608.00814#bib.bib13); Tanget al\.[2023](https://arxiv.org/html/2608.00814#bib.bib14); Patilet al\.[2025](https://arxiv.org/html/2608.00814#bib.bib15); Llama Team[2024](https://arxiv.org/html/2608.00814#bib.bib19)\)\. Across all 21 target–workload combinations,OoO\-Specis the fastest applicable method and reaches up to5\.34×5\.34\\timesspeedup over autoregressive decoding; its mean speedup is3\.89×3\.89\\times, compared with2\.95×2\.95\\timesforToolSpec\. Against the released learned drafters—EAGLE\-3, PARD\-2, and DFlash—OoO\-Specleads in every comparable cell across Llama\-3\.1\-8B and Qwen3\-4B/8B/14B\. With the same sidecar, the gain overToolSpecdoes not shrink as the target grows: across Qwen3 4B, 8B, 14B, and 32B,OoO\-Spec’s per\-target overall speedup exceedsToolSpec’s by 27\.1%, 31\.0%, 40\.9%, and 37\.3%, averaging 34\.1%\.

Our contributions are:

- •We introduce Out\-of\-Order semantic Speculation for tool calls: a sidecar computes schema\-defined function and argument values ahead of their textual positions while the target continues to commit tokens in order\.
- •We make the sidecar’s proposal usable whenever it becomes ready: every candidate\-construction boundary inToolSpecis a join opportunity, so a proposal that arrives mid\-generation still accelerates the tokens not yet committed\.
- •We show that one Qwen3\-0\.6B sidecar transfers across target sizes, model families, and three tool\-use workloads without per\-target drafter training, with sidecar and communication costs counted in end\-to\-end time\.

## 2Related Work

#### Learned speculative decoding\.

Classical speculative decoding uses a smaller autoregressive model to draft a token continuation that is subsequently verified by the target\(Leviathanet al\.[2023](https://arxiv.org/html/2608.00814#bib.bib1); Chenet al\.[2023](https://arxiv.org/html/2608.00814#bib.bib2)\)\. Later methods reduce the serial depth of drafting\. Medusa predicts several future positions with additional decoding heads, while EAGLE and EAGLE\-3 construct proposals from target\-aligned features\(Caiet al\.[2024](https://arxiv.org/html/2608.00814#bib.bib3); Liet al\.[2024](https://arxiv.org/html/2608.00814#bib.bib4),[2025](https://arxiv.org/html/2608.00814#bib.bib5)\)\. PARD\-2 and DFlash further parallelize block drafting with target\-aligned or diffusion\-style drafters\(Anet al\.[2026](https://arxiv.org/html/2608.00814#bib.bib10); Chenet al\.[2026](https://arxiv.org/html/2608.00814#bib.bib11)\), and semantic\-aware variants probe the target’s internal states\(Donget al\.[2026](https://arxiv.org/html/2608.00814#bib.bib12)\), again bound to one target’s representations\. All of them, however, require training a dedicated drafter for every target model they serve, at substantial cost\. In contrast,OoO\-Specdrafts schema\-defined semantic strings, allowing one frozen sidecar to serve multiple targets without target\-specific retraining\.

#### Train\-free and retrieval\-based drafting\.

Another line of work avoids training a target\-specific drafter by reusing tokens already available at inference time\. Prompt Lookup Decoding retrieves repeated prompt spans, Token Recycling builds candidates from observed token transitions, and SAM\-Decoding uses a suffix automaton to retrieve continuations\(Saxena[2023](https://arxiv.org/html/2608.00814#bib.bib8); Luoet al\.[2025](https://arxiv.org/html/2608.00814#bib.bib6); Huet al\.[2024](https://arxiv.org/html/2608.00814#bib.bib7)\)\.ToolSpecspecializes this approach to function calls through schema\-driven drafting and History\-Calls retrieval\(Xiaet al\.[2026](https://arxiv.org/html/2608.00814#bib.bib9)\), described in Section[3](https://arxiv.org/html/2608.00814#S3)\. Effective as these sources are, none of them can supply a request\-specific value that has never appeared in the schema, the context, or the history\.

## 3Preliminaries

![Refer to caption](https://arxiv.org/html/2608.00814v1/x2.png)Figure 2:Online inference inOoO\-Spec\. The sidecar’s slot predictions are joined and rendered into a hint bank that entersToolSpec’s retrieval at candidate\-construction boundaries without blocking the target; the target verifier alone commits output tokens\.#### Target authority\.

Letxxbe a request andy<ty\_\{<t\}the prefix already committed by the target\. Under greedy decoding, the next committed token is

yt=arg⁡maxv⁡pT​\(v∣x,y<t\)\.y\_\{t\}=\\arg\\max\_\{v\}p\_\{T\}\(v\\mid x,y\_\{<t\}\)\.\(1\)A speculative method changes which continuations are evaluated together, affecting the number of tokens accepted per step and the resulting latency\. Only the target extendsy<ty\_\{<t\}and determines the output\.

#### TheToolSpecdecoding loop\.

ToolSpecspecializes target\-verified speculation to function calls\(Xiaet al\.[2026](https://arxiv.org/html/2608.00814#bib.bib9)\)\. It maintains two proposal sources\. A finite\-state machine compiled from the tool schemas tracks the current region of the call and supplies constrained scaffolding such as field names and delimiters\. History Calls provide retrieved continuations for variable regions\. At each round,ToolSpeccombines both sources into a candidate tree and evaluates it with the target\. The target commits the longest approved prefix\. If decoding continues,ToolSpecbuilds the next tree from all tokens committed so far\.

We call each point at which a new tree is built a*candidate\-construction boundary*\. These boundaries recur throughout a call\. A later round can therefore include candidates that were unavailable when decoding began, while tokens already committed by the target remain unchanged\. Our method uses this recurring construction step to add one candidate source whose proposals may arrive after decoding starts\.ToolSpeccontinues to control the FSM, History Calls, candidate budget, packed verification, KV\-cache updates, and recovery\.

## 4OoO\-Spec

OoO\-Specconsists of a shared semantic sidecar and an asynchronous inference loop that connects it to the target\. The sidecar is trained once and reused across target models\. At inference time, it predicts the function choice and all schema\-defined argument slots in a single parallel batch, concurrently with the target’sToolSpecloop\. The target checks the resulting hint at eligible candidate\-construction boundaries without blocking and remains responsible for verification and left\-to\-right token commitment\. We first describe sidecar training and then the two online inference paths\.

### 4\.1Training a Shared Semantic Sidecar

The supplementary material diagrams the offline training pipeline and lists the frozen asset identities\.

#### Training requests\.

We construct fixed training and development splits from API\-Bank and ToolAlpaca\(Liet al\.[2023](https://arxiv.org/html/2608.00814#bib.bib13); Tanget al\.[2023](https://arxiv.org/html/2608.00814#bib.bib14)\): 6,200 training and 335 development requests from API\-Bank, and 3,462 training and 194 development requests drawn only from ToolAlpaca’s training file\. ToolAlpaca is decontaminated by removing an entire API whenever its API name or any function name appears in the evaluation inventory, and we use neither its golden answers nor its tool\-execution outputs\. Both datasets use a deterministic prompt\-hash split, giving 9,662 training and 529 development source requests with zero request\-ID or prompt\-hash overlap between them\.

#### Teacher traces\.

We fix Qwen2\.5\-32B\-Instruct as the offline teacher\(Qwen Team[2024](https://arxiv.org/html/2608.00814#bib.bib17)\)\. It receives each dialogue together with its tool schemas and generates a structured call greedily\. We parse supported tool\-call surface forms into a strict\-JSON domain and retain the first valid call, keeping only the normalized function name and argument map; teacher token IDs, chat\-template tokens, and tool\-call control tokens are discarded\.

#### Slot supervision\.

Each retained semantic call is expanded into three task types under the Qwen3\-0\.6B chat template\. A*function\-index*row asks for the selected function’s local index in the current schema\. An*argument\-value\-or\-null*row is created for every parameter of every candidate function; its target is the compact JSON value when the parameter belongs to the selected function and appears in the teacher call, andnullotherwise\. A third, auxiliary*direct\-call*row asks for the complete normalized call\. The expansion produces 75,567 training and 3,239 development rows\. The online sidecar queries only function\-index and argument rows\.

#### Optimization and freezing\.

We initialize an unadapted Qwen3\-0\.6B\(Yanget al\.[2025](https://arxiv.org/html/2608.00814#bib.bib18)\)and train one LoRA adapter\(Huet al\.[2022](https://arxiv.org/html/2608.00814#bib.bib16)\)for a single epoch, masking prompt labels so the objective is causal cross\-entropy over completion tokens; the supplementary material lists the full hyperparameters\. After the fixed development pass we freeze the epoch\-one adapter\. The same Qwen3\-0\.6B base and adapter serve every target and workload, so deployment to a new target requires no target\-specific gradient update, and the teacher is absent at inference time\.

### 4\.2Asynchronous Main Inference Loop

With the shared adapter frozen, online inference follows the two paths in Figure[2](https://arxiv.org/html/2608.00814#S3.F2)\. At request arrival, the runtime submits the complete sidecar job to a background future and immediately starts target prefill\. In the reported split configuration, the target and sidecar run as separate processes on two GPUs, and decoded slot strings travel over loopback HTTP; colocated execution is used only as an ablation\. One batched sidecar wave is launched per request, whereas the target repeatedly constructs and verifies candidate trees\. A completed future can affect candidate construction only at boundaries whereToolSpecinvokes its retrieval finder\. These checks observe availability and never stall target execution\.

We use*sidecar path*to include the background response processing that follows model generation\. The learned sidecar itself produces only slot answers\. Parsing, deterministic assembly, rendering, and target tokenization remain runtime operations\.

### 4\.3Sidecar Path: From Schemas to a Hint Bank

#### Slot construction\.

Letℱ\\mathcal\{F\}be the candidate functions in a request andPfP\_\{f\}the parameters of functionff\. The runtime constructs the slot set

𝒮\\displaystyle\\mathcal\{S\}=\{R\}∪\{Af,p:f∈ℱ,p∈Pf\},\\displaystyle=\\\{R\\\}\\cup\\\{A\_\{f,p\}:f\\in\\mathcal\{F\},\\ p\\in P\_\{f\}\\\},\(2\)\|𝒮\|\\displaystyle\|\\mathcal\{S\}\|=1\+∑f∈ℱ\|Pf\|\.\\displaystyle=1\+\\sum\_\{f\\in\\mathcal\{F\}\}\|P\_\{f\}\|\.RRasks for the local function index\. EachAf,pA\_\{f,p\}identifies one candidate function and parameter and asks for a compact JSON value ornull\. Every prompt contains the dialogue and indexed schemas\. In particular, an argument prompt does not receive the answer toRR\.

#### Batched prediction\.

All slots are submitted together as one request\-level vLLM\(Kwonet al\.[2023](https://arxiv.org/html/2608.00814#bib.bib20)\)wave, with greedy generation and at most 32 new tokens per slot\. Waiting forRRbefore issuing the selected function’s arguments would add a serial stage\. Instead, argument slots for unselected functions are computed speculatively alongside the selected branch\.

#### Join and render\.

After all slot answers return, the response path extracts the first decodable JSON value from each answer\. It maps the function value to a legal local index under a deterministic fallback, keeps non\-null argument values for the selected function, and discards unavailable or malformed values\. The runtime then joins the function name and remaining values with schema\-supplied field names and JSON structure to form one normalized object\. An object or array parameter remains one JSON\-valued slot, and the online path does not issue a separate whole\-call prompt\. The normalized object is rendered under a small frozen policy into equivalent textual views\. Together, these views represent one semantic prediction, not multiple predicted calls\. Qwen3 targets use fixed plain\-JSON, Markdown, and XML\-style views\. For Qwen2\.5 and Llama targets, the layout policy is frozen from training and development traces before formal evaluation\. This policy is target\-specific configuration and requires no gradient update\. Raw sidecar token IDs are discarded, so the device boundary carries decoded slot strings and the target retains control of its own tokenization\.

### 4\.4Target Path: Prefill and Nonblocking Hint Pickup

#### Prefill\.

The target begins prefill as soon as the request arrives and then enters its nativeToolSpecloop\. Schema\-FSM rounds remain unchanged regardless of the sidecar state\. While the future is pending, retrieval rounds use the original History Calls and fallback candidates\.

#### Candidate boundary\.

At a retrieval\-candidate roundtt, the runtime performs a nonblocking readiness check and forms

𝒞t=\{𝒞tTS,sidecar future pending,MergeB⁡\(𝒞thint,𝒞tTS\),hint bank ready,\\mathcal\{C\}\_\{t\}=\\begin\{cases\}\\mathcal\{C\}^\{\\mathrm\{TS\}\}\_\{t\},&\\text\{sidecar future pending\},\\\\ \\operatorname\{Merge\}\_\{B\}\(\\mathcal\{C\}^\{\\mathrm\{hint\}\}\_\{t\},\\mathcal\{C\}^\{\\mathrm\{TS\}\}\_\{t\}\),&\\text\{hint bank ready\},\\end\{cases\}\(3\)where𝒞tTS\\mathcal\{C\}^\{\\mathrm\{TS\}\}\_\{t\}denotes the nativeToolSpeccandidates for that retrieval round andMergeB\\operatorname\{Merge\}\_\{B\}preserves its fixed candidate budget\. A future that completes just after one check is first observed at a later retrieval boundary; the target performs no rollback\.

#### Tokenizer and hint bank\.

The target\-side bridge tokenizes each rendered view once, inserts the target EOS token between views, and concatenates them into one request\-level hint bank\. Encoding and device materialization occur once, and later retrieval rounds reuse the same bank\. Readiness is queried only at retrieval\-candidate rounds\.

#### Suffix matching\.

To retrieve a continuation for the target’s current position, letst=suffixn⁡\(y<t\)s\_\{t\}=\\operatorname\{suffix\}\_\{n\}\(y\_\{<t\}\)be the length\-nnsuffix of its committed prefix\. The finder searches the encoded bank for exact occurrences ofsts\_\{t\}, tryingn=7n=7down to11, and proposes the tokens following a match\. Matching the current suffix prevents a late hint from proposing content for a position that the target has already committed\. Hint continuations fill the availableToolSpeclanes first, and its native sources fill the remaining lanes\. If the bank has no aligned continuation, the round uses only native candidates\. Repeating this alignment at later retrieval boundaries also lets one request\-level bank remain useful after an earlier field has already been generated or rejected\.

### 4\.5Target\-Only Verification and Recovery

#### Verify and commit\.

Hint continuations and nativeToolSpeccontinuations are compiled into the same candidate tree\. The target evaluates the tree in one forward pass and commits the longest prefix allowed by its greedy decision rule\. At the first mismatch, it rejects the remaining candidate suffix and uses its own next token to continue decoding\. Only states on the accepted target path enter the committed KV cache; sidecar outputs never update that cache or commit tokens\.

#### Recovery\.

Rejecting one continuation leaves the rest of the request\-level bank available for later suffix matches\. An inaccurate hint can consume candidate or verification work, and a hint that finishes after target completion is ignored\. In every case, the target determines the committed output\.OoO\-Specchanges candidate source and availability;ToolSpecand the target retain candidate\-tree construction, verification, cache updates, and recovery\.

## 5Experimental Setup

### 5\.1Targets, Workloads, and Protocol

The fully ranked comparison uses seven targets: Qwen2\.5\-7B/14B\(Qwen Team[2024](https://arxiv.org/html/2608.00814#bib.bib17)\), Llama\-3\.2\-3B and Llama\-3\.1\-8B\(Llama Team[2024](https://arxiv.org/html/2608.00814#bib.bib19)\), and Qwen3\-4B/8B/14B\(Yanget al\.[2025](https://arxiv.org/html/2608.00814#bib.bib18)\); Qwen3\-32B and Qwen2\.5\-32B join the scaling study\. Every method sees the same 597 API\-Bank\(Liet al\.[2023](https://arxiv.org/html/2608.00814#bib.bib13)\), 193 ToolAlpaca\(Tanget al\.[2023](https://arxiv.org/html/2608.00814#bib.bib14)\), and 68 BFCL Java/JavaScript\(Patilet al\.[2025](https://arxiv.org/html/2608.00814#bib.bib15)\)requests; the BFCL set is drawn from the v4 Java and JavaScript categories\. Table[1](https://arxiv.org/html/2608.00814#S5.T1)summarizes the three evaluation sets\. Tool repetition falls from 8\.91 on API\-Bank to 1\.00 on BFCL, where every request exposes a distinct tool, so history retrieval has the least to reuse there\.

Table 1:Evaluation\-set statistics\. \#Tools counts distinct tools in the request prompts; Avg\. Tool Rep\. is golden invocations per invoked tool, followingXiaet al\.\([2026](https://arxiv.org/html/2608.00814#bib.bib9)\)\.All measurements use greedy decoding, batch size one, and H100\-80GB GPUs, with generation limits fixed across methods per benchmark\. Qwen2\.5 and Llama targets runToolSpec’s FP16 cache backend; Qwen3 targets use the architecture\-specific BF16 Transformers\(Wolfet al\.[2020](https://arxiv.org/html/2608.00814#bib.bib21)\)port\.OoO\-Specalways places the target and the sidecar on two separate GPUs\. Reported request latency is wall time from before draft submission until both target and sidecar have finished, so sidecar launch, communication, and coordination are all inside the measurement\.

### 5\.2Baselines and Placement

We compare live AR \(Vanilla\), Prompt Lookup Decoding \(PLD\), Token Recycling \(TR\), SAM\-Decoding \(SAMD\), andToolSpecon every target, each through its native proposal and verification path with its published configuration\. Where an author\-supported checkpoint exists we add EAGLE\-3, PARD\-2, and DFlash; each learned drafter uses its faster measured eligible placement \(EAGLE\-3 split; PARD\-2 and DFlash their official colocated Transformers/Transformers\+ paths\)\.

### 5\.3Metrics and Correctness Audit

FollowingToolSpec, each cell reports mean accepted tokens per target verification call \(\#MAT\), mean per\-request tokens per second, and elapsed\-time speedup against the live AR reference for the same target and workload; the overall column is the unweighted mean across the three benchmarks\. Speed is interpreted only after correctness: for every request we compare the method’s output tokens with the live greedy AR trajectory of the same runner\.

## 6Results

Table 2:Main comparison under greedy decoding; bold marks the best value per metric\.OoO\-Specalways uses the same frozen split sidecar\.∗Learned drafters use their faster measured official placement \(EAGLE\-3 split; PARD\-2 and DFlash colocated\); their rows appear only where an author\-released checkpoint exists\.### 6\.1Main Comparison

OoO\-Specis the fastest method in every cell it is measured in\. Table[2](https://arxiv.org/html/2608.00814#S6.T2)reports the complete matrix: across all seven targets and three benchmarks,OoO\-Specleads in 21 of 21 cells, spans2\.46×2\.46\\times–5\.34×5\.34\\timesover live AR, and averages3\.89×3\.89\\timesagainst2\.95×2\.95\\timesforToolSpec\. On the Qwen2\.5 and Llama targets its overall speedups are3\.66×3\.66\\times,3\.57×3\.57\\times,3\.02×3\.02\\times, and3\.23×3\.23\\times, where the strongest prior methods reach2\.83×2\.83\\times,2\.64×2\.64\\times,2\.57×2\.57\\times, and2\.84×2\.84\\times; on Qwen3\-4B, 8B, and 14B it reaches4\.46×4\.46\\times,4\.73×4\.73\\times, and4\.55×4\.55\\times\. Against each cell’s strongest prior method, the per\-target margin ranges from 7\.7% \(Qwen3\-4B\) to 40\.9% \(Qwen3\-14B\), averaging 23\.4%\.

Accepted length alone does not decide these rankings\. On Llama\-3\.1\-8B, PARD\-2 accepts more tokens per verification call thanOoO\-Specon ToolAlpaca \(5\.15 vs\. 3\.86\) and BFCL \(5\.29 vs\. 3\.95\), yet reaches only2\.18×2\.18\\timesand1\.78×1\.78\\timeswhereOoO\-Specreaches2\.59×2\.59\\timesand2\.74×2\.74\\times\. Drafting exposure, verifier cost, and runtime overhead all sit inside request latency;OoO\-Specwins not by accepting the most tokens but by adding accepted length at almost no exposed cost\.

### 6\.2Transfer Without Target\-Specific Training

EveryOoO\-Specnumber in this paper comes from one adapter, and the transfer happens along three axes at once\. Across families and tokenizers: the sidecar is a Qwen3\-0\.6B trained on Qwen\-generated semantic text, yet its largest single\-benchmark speedup on a non\-Qwen target is4\.36×4\.36\\times\(Llama\-3\.1\-8B, API\-Bank\), because the target re\-tokenizes every hint and no Qwen token ID ever reaches a Llama model\. Across target scales: the same 0\.6B checkpoint serves 3B through 32B targets; it predicts schema slots, not target hidden states, so nothing about it refers to target width or depth\. Across workloads: API\-Bank, ToolAlpaca, and BFCL Java/JavaScript share the checkpoint, and BFCL was never represented in training at all\.

The contrast with learned drafters is practical: every EAGLE\-3, PARD\-2, or DFlash result depends on a per\-target training artifact, and a target without an author\-released checkpoint cannot be served at all, whereas the sidecar asked nothing of any target but its tokenizer\.

### 6\.3Scaling to Larger Targets

![Refer to caption](https://arxiv.org/html/2608.00814v1/x3.png)Figure 3:Qwen3 target scaling with the frozen sidecar\. ColocatedOoO\-Specstays close toToolSpec, while split placement sustains4\.46×4\.46\\times–4\.75×4\.75\\times; the colocated curve ends at 14B, where colocation no longer fits one GPU\.Figure[3](https://arxiv.org/html/2608.00814#S6.F3)holds the sidecar fixed and grows the target from Qwen3\-4B to 32B\.OoO\-Specaverages4\.46×4\.46\\times,4\.73×4\.73\\times,4\.55×4\.55\\times, and4\.75×4\.75\\timesagainstToolSpec’s3\.51×3\.51\\times,3\.61×3\.61\\times,3\.23×3\.23\\times, and3\.46×3\.46\\times—relative improvements of 27\.1%, 31\.0%, 40\.9%, and 37\.3%, or 34\.1% on average\. The direction is what the schedule predicts\. The sidecar’s work is a small structured object whose cost barely depends on the target; the target’s per\-step cost grows with its parameters; so every accepted semantic continuation displaces increasingly expensive verification steps\. The same frozen sidecar also reaches an overall4\.58×4\.58\\timeson Qwen2\.5\-32B\.

### 6\.4Placement Ablation

Figure[3](https://arxiv.org/html/2608.00814#S6.F3)includes a placement ablation: the same frozen sidecar colocated on the target’s GPU, measured in a separate run against its own AR reference\. Colocated overall speedups are3\.54×3\.54\\times,3\.50×3\.50\\times, and3\.32×3\.32\\timeson Qwen3\-4B, 8B, and 14B—within0\.11×0\.11\\timesofToolSpec—and Qwen3\-32B cannot host both models on a single 80 GB GPU\. The ablation confirms the value of split placement: with the sidecar on its own device, the same frozen hints lift overall speedup fromToolSpec\-level to4\.46×4\.46\\times–4\.75×4\.75\\times, consistent with the critical\-path analysis of Section[6\.5](https://arxiv.org/html/2608.00814#S6.SS5)\.

### 6\.5Sidecar Latency and Communication

Aggregated over 4,290 measured requests, the target path averages 309\.5 ms per request and the sidecar 85\.0 ms of wall time\. The two run concurrently, so these numbers do not add: observedOoO\-Specend\-to\-end latency is 311\.9 ms, only 2\.4 ms above the target path alone\. Nearly all sidecar work hides behind target decoding\.

The sidecar’s 85\.0 ms is almost entirely generation \(82\.2 ms\)\. The remaining 2\.79 ms covers local transport, response processing, JSON decoding, and scheduling together, and thus bounds from above the full cost of delivering a hint\. The semantic payload it delivers averages 85 bytes per request excluding protocol metadata\. Communication inOoO\-Specis therefore both infrequent \(once per request\) and small \(tens of bytes\), which is why device separation is cheap\.

Timing also confirms that there is enough target work to overlap: 65\.2% of hints are ready before the first candidate construction, 98\.6% before the target finishes, and 94\.6% of hints are eventually used\. The asynchronous design is not waiting on a rare fast path; on the large majority of requests the hint arrives early enough to matter and is actually consumed\.

## 7Discussion

#### Why split placement helps\.

OoO\-Specbenefits from a second GPU for two reasons\. Semantic drafting is asynchronous—the sidecar works from the request and schema without following the target’s current token position—and the separation needs little communication: one compact semantic result per request rather than target\-conditioned states throughout decoding\. Our system analysis \(Section[6\.5](https://arxiv.org/html/2608.00814#S6.SS5)\) shows this payload and its response overhead are small next to either model’s execution, so most proposal work overlaps without a high\-bandwidth, step\-synchronous device boundary\.

#### Deployment implications\.

Because target and sidecar share no model dimensions, token IDs, or device\-local tensors, heterogeneous placement is possible in principle: scarce high\-memory accelerators reserved for the target, semantic drafting on a lower\-cost device, and independent scaling of both\. One sidecar service might also batch requests from multiple target replicas, amortizing its model memory; evaluating heterogeneous hardware and multi\-target serving is left to future work\.

#### On\-device tool use\.

On\-device agents are another natural setting: edge machines often pair an integrated GPU with a discrete one, or two GPUs of very different capability, behind interconnects far weaker than a server fabric\.OoO\-Speccould keep the target on the stronger device and draft semantics on the smaller one, since the devices exchange a request\-level semantic result rather than step\-synchronous hidden states; quantifying latency, energy, and concurrency on edge hardware remains future work\.

## 8Conclusion

Tool calls are committed in textual order, but nothing forces their semantics to be computed that way\.OoO\-Specresolves a call’s function and argument slots out of order on a small frozen sidecar, hands them to the target as re\-tokenizable text, and letsToolSpec’s recurring candidate constructions absorb them whenever they are ready—while the target verifies and commits every token in order, exactly as before\. One 0\.6B sidecar, trained once, is the fastest method in all 21 ranked target–benchmark cells\. Decoupling when a proposal is generated from when it is used gives a simple, transferable recipe for faster tool calling\.

## References

- Z\. An, T\. Liu, Z\. Liu, D\. Li, R\. Liu, and E\. Barsoum \(2026\)PARD\-2: target\-aligned parallel draft model for dual\-mode speculative decoding\.arXiv preprint arXiv:2605\.08632\.Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p2.1),[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. Dao \(2024\)Medusa: simple LLM inference acceleration framework with multiple decoding heads\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. Jumper \(2023\)Accelerating large language model decoding with speculative sampling\.arXiv preprint arXiv:2302\.01318\.Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p2.1),[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Chen, Y\. Liang, and Z\. Liu \(2026\)DFlash: block diffusion for flash speculative decoding\.InProceedings of the 43rd International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p2.1),[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Dong, S\. Wang, D\. Lin, B\. Chen, and A\. E\. Hassan \(2026\)Beyond tokens: semantic\-aware speculative decoding for efficient inference by probing internal states\.arXiv preprint arXiv:2602\.03708\.Cited by:[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.00814#S4.SS1.SSS0.Px4.p1.1)\.
- Y\. Hu, K\. Wang, X\. Zhang, F\. Zhang, C\. Li, H\. Chen, and J\. Zhang \(2024\)SAM Decoding: speculative decoding via suffix automaton\.arXiv preprint arXiv:2411\.10666\.Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p2.1),[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,Cited by:[§4\.3](https://arxiv.org/html/2608.00814#S4.SS3.SSS0.Px2.p1.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InProceedings of the 40th International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p2.1),[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. Li \(2023\)API\-Bank: a comprehensive benchmark for tool\-augmented LLMs\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p6.3),[§4\.1](https://arxiv.org/html/2608.00814#S4.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.00814#S5.SS1.p1.1)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2024\)EAGLE: speculative sampling requires rethinking feature uncertainty\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2025\)EAGLE\-3: scaling up inference acceleration of large language models via training\-time test\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p2.1),[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px1.p1.1)\.
- Llama Team \(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p6.3),[§5\.1](https://arxiv.org/html/2608.00814#S5.SS1.p1.1)\.
- X\. Luo, Y\. Wang, Q\. Zhu, Z\. Zhang, X\. Zhang, Q\. Yang, and D\. Xu \(2025\)Turning trash into treasure: accelerating inference of large language models with token recycling\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,pp\. 6816–6831\.Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p2.1),[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px2.p1.1)\.
- S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez \(2025\)The berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InProceedings of the 42nd International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p6.3),[§5\.1](https://arxiv.org/html/2608.00814#S5.SS1.p1.1)\.
- Qwen Team \(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.00814#S4.SS1.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2608.00814#S5.SS1.p1.1)\.
- A\. Saxena \(2023\)Prompt lookup decoding\.Note:GitHub repository,https://github\.com/apoorvumang/prompt\-lookup\-decodingCited by:[§1](https://arxiv.org/html/2608.00814#S1.p2.1),[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Tang, Z\. Deng, H\. Lin, X\. Han, Q\. Liang, B\. Cao, and L\. Sun \(2023\)ToolAlpaca: generalized tool learning for language models with 3000 simulated cases\.arXiv preprint arXiv:2306\.05301\.Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p6.3),[§4\.1](https://arxiv.org/html/2608.00814#S4.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.00814#S5.SS1.p1.1)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz,et al\.\(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 38–45\.Cited by:[§5\.1](https://arxiv.org/html/2608.00814#S5.SS1.p2.1)\.
- H\. Xia, Y\. Li, C\. Du, M\. Song, and W\. Li \(2026\)ToolSpec: accelerating tool calling via schema\-aware and retrieval\-augmented speculative decoding\.arXiv preprint arXiv:2604\.13519\.Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p2.1),[§2](https://arxiv.org/html/2608.00814#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.00814#S3.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2608.00814#S5.T1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2608.00814#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.00814#S4.SS1.SSS0.Px4.p1.1),[§5\.1](https://arxiv.org/html/2608.00814#S5.SS1.p1.1)\.