AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

# Action and Observation Co-Speculation for Low-Latency Agent Serving
Source: [https://arxiv.org/html/2608.00881](https://arxiv.org/html/2608.00881)
###### Abstract

Large language model agents increasingly act through stateful tools, yet model generation and environment execution remain serialized at every step\. As decoding accelerates, tool execution becomes a growing bottleneck\. Existing action\- or observation\-only speculation leaves much of this latency exposed: value is concentrated in a few slow calls, some outcomes emerge only through execution, and longer lookahead typically requires an increasingly unlikely chain of action predictions\. We presentAOSpec, a lossless framework that co\-speculates actions and observations across the full agent–environment loop\. Expected Value Decoding \(EVD\) directs observation speculation toward outcomes with the greatest expected latency benefit, optimizing expected time hidden rather than hit rate\. For outcomes only execution can reveal,AOSpeclaunches latency\-critical target actions in isolated forks that contain their effects, while Joint Action–State Verification \(JASV\) verifies both the action and its origin state against committed execution before reuse\. JASV recasts long\-horizon action dependency from full\-chain prediction into target action–state verification, breaking the lookahead–accuracy tradeoff and unlocking long\-range overlap without sacrificing serial semantics\. Across Terminal\-Bench serving settings spanning four harnesses, five actor models, and five serving speeds,AOSpecoutperforms every practical baseline, reducing mean end\-to\-end latency by 11\.8–32\.5% and p99 latency by up to 42\.8%\. Its gains increase as decoding accelerates, and its observation model transfers from Terminal\-Bench to SWE\-bench Verified without retraining\.

## 1Introduction

Modern large language model \(LLM\)\-based agents interact with external environments through iterative cycles of reasoning, action, and observation\(Yao et al\.[2022](https://arxiv.org/html/2608.00881#bib.bib27)\)\. They increasingly rely on stateful tools that inspect or modify evolving runtime environments, extending beyond simple information retrieval\. Frameworks such as Claude Code and OpenClaw decouple model serving from the tool\-execution environment\(Anthropic[2026](https://arxiv.org/html/2608.00881#bib.bib1); OpenClaw Foundation[2026](https://arxiv.org/html/2608.00881#bib.bib17)\), forming what we term a disaggregated agent loop\. Because the standard loop serializes model generation and tool execution, tools account for a growing share of end\-to\-end latency as decoding accelerates\(Tile\-AI[2026](https://arxiv.org/html/2608.00881#bib.bib23); Artificial Analysis[2026a](https://arxiv.org/html/2608.00881#bib.bib2),[b](https://arxiv.org/html/2608.00881#bib.bib3); Groq[2024](https://arxiv.org/html/2608.00881#bib.bib11); Cerebras Systems[2025](https://arxiv.org/html/2608.00881#bib.bib7)\)\.

Speculation can break this serialization by starting likely future work early\. Existing systems target one side of the loop:\(i\)action speculation predicts and launches future tool calls\(Ye et al\.[2025](https://arxiv.org/html/2608.00881#bib.bib28); Sui et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib21); Nichols et al\.[2025](https://arxiv.org/html/2608.00881#bib.bib16); Song[2026](https://arxiv.org/html/2608.00881#bib.bib20); Bai et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib4)\), whereas\(ii\)observation speculation continues generation from provisional observations verified asynchronously\(Saberi, Rezaei, and Feizi[2026](https://arxiv.org/html/2608.00881#bib.bib19)\)\.

However, existing speculation methods fall short on heterogeneous stateful tool workloads, exposing three barriers to effective speculation\.\(1\) Latency concentration:a few slow calls dominate tool time, decoupling hit rate from latency savings \(Section[3\.1](https://arxiv.org/html/2608.00881#S3.SS1)\)\.\(2\) Environment dependence:some environment\-dependent observations emerge only through execution, making observation prediction alone insufficient and requiring sandboxed action speculation \(Section[3\.2](https://arxiv.org/html/2608.00881#S3.SS2)\)\.\(3\) Lookahead tradeoff:launching speculative actions earlier increases their overlap with model generation, but lossless generation requires the intervening action chain to match; compounding errors collapse accuracy and erase the potential speedup \(Section[3\.3](https://arxiv.org/html/2608.00881#S3.SS3)\)\. Together, these findings demand a new approach to speculation across the full agent–environment loop\.

We presentAOSpec, a lossless framework that jointly speculates actions and observations across the disaggregated agent loop \(Section[4](https://arxiv.org/html/2608.00881#S4)\)\. To address latency concentration,*Expected Value Decoding*\(EVD\) generates observation candidates by combining their probability and estimated tool time, directly optimizing expected time hidden rather than hit rate\. For observations only execution can reveal,AOSpeclaunches latency\-critical target actions in isolated forks that contain their effects\(Tencent Cloud[2026](https://arxiv.org/html/2608.00881#bib.bib22); Dong et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib9)\)\. To break the lookahead–accuracy tradeoff,*Joint Action–State Verification*\(JASV\) recasts long\-horizon action dependency from full\-chain prediction into target action–state verification: a fork is verified and selected only when its predicted action and origin state match the emitted action and committed state\. This unlocks long\-range overlap without predicting intervening actions, while exact verification preserves serial semantics and lossless\. Together, these mechanisms yield 11\.8–32\.5% end\-to\-end latency savings, with gains increasing as decoding accelerates as shown in Figure[1](https://arxiv.org/html/2608.00881#S3.F1)\.

We make three contributions:

- •We characterize and quantify the latency concentration, information boundary, and lookahead tension that limit speculation over stateful tools\.
- •We introduceAOSpec, combining EVD for latency\-aware observation drafting, isolated action execution for environment\-dependent observations, and JASV for lossless long\-range lookahead without full\-chain prediction\.
- •Across 4 different harnesses and 5 models on Terminal\-Bench,AOSpecachieves 11\.8–32\.5% end\-to\-end latency savings, reduces p99 latency by up to 42\.8% and outperforms practical baselines\.

## 2Related Work

##### Speculative execution for LLM agents\.

Speculative decoding accelerates generation through draft\-and\-verify execution\(Cai et al\.[2024](https://arxiv.org/html/2608.00881#bib.bib5); Li et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib14); Chen et al\.[2024](https://arxiv.org/html/2608.00881#bib.bib8)\)\. Agent systems extend this principle by pre\-executing predicted actions or tool calls\(Ye et al\.[2025](https://arxiv.org/html/2608.00881#bib.bib28); Song[2026](https://arxiv.org/html/2608.00881#bib.bib20); Nichols et al\.[2025](https://arxiv.org/html/2608.00881#bib.bib16); Bai et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib4); Zhong et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib29); Sui et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib21)\), or by continuing from provisional observations\(Saberi, Rezaei, and Feizi[2026](https://arxiv.org/html/2608.00881#bib.bib19); Cao et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib6)\)\. Unlike prior approaches,AOSpecjointly optimizes both forms in a lossless framework designed for stateful runtime tools with highly skewed latency, environment\-dependent outputs, and side effects\.

##### Agent sandboxes\.

Agent sandboxes such as CubeSandbox and DeltaBox isolate and revert file and process effects, enabling speculation over side\-effecting tools\(Tencent Cloud[2026](https://arxiv.org/html/2608.00881#bib.bib22); Dong et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib9)\)\.AOSpecuses these primitives to implement its isolation contract, while effects that cannot be isolated remain ineligible\.

## 3Motivation

![Refer to caption](https://arxiv.org/html/2608.00881v1/x1.png)Figure 1:End\-to\-end latency savings under trace replay, averaged over nine \(harness, model\) configurations on Terminal Bench\. Left: ceiling gaps among the Spec\. Act, Spec\. Act w/ lookahead, Spec\. Obs, andAOSpec\. Right:AOSpecversus SpecHop at 20/10/1 ms per token\.We model a disaggregated agent as an actor and an environment runtime\. At stepii, the actor takesDiD\_\{i\}time to generate actionaia\_\{i\}, and the runtime takesTiT\_\{i\}time to execute it and return observationoio\_\{i\}, yielding serial latency∑i\(Di\+Ti\)\\sum\_\{i\}\(D\_\{i\}\+T\_\{i\}\)\. Observation speculation draftso^i\\hat\{o\}\_\{i\}so the actor can advance whileaia\_\{i\}executes\. Action speculation drafts and begins executinga^i\\hat\{a\}\_\{i\}beforeaia\_\{i\}is emitted\. By profiling 1,921 training\-set trajectories across nine Terminal Bench harness–model configurations\(Merrill et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib15)\), we identify three fundamental challenges: latency concentration, environment\-dependent observations, and the tension between longer overlap and compounding chain\-prediction error\.

![Refer to caption](https://arxiv.org/html/2608.00881v1/x2.png)

\(a\) Concentration

![Refer to caption](https://arxiv.org/html/2608.00881v1/x3.png)

\(b\) Latency scatter

![Refer to caption](https://arxiv.org/html/2608.00881v1/x4.png)

\(c\) Qualitative examples

![Refer to caption](https://arxiv.org/html/2608.00881v1/x5.png)

\(d\) Overlap ceiling

Figure 2:Motivating measurements from Terminal Bench\. \(a\) A small fraction of long calls accounts for almost all tool time\. \(b\) Per\-step latency spans both model\-bound and tool\-bound regimes\. \(c\) Qualitative examples of costly environment\-dependent observations\. \(d\) Longer lookahead raises the oracle overlap ceiling, but empirical exact\-chain hit rate collapses\.### 3\.1Latency Concentration Decouples Hits from Savings

As shown in Figure[2](https://arxiv.org/html/2608.00881#S3.F2)\(a\), tool time of computer\-using agents is highly concentrated: the 17% of calls lasting at least one second account for 97% of total tool time\. Figure[2](https://arxiv.org/html/2608.00881#S3.F2)\(b\) shows the same mismatch at the step level: execution spans both model\-bound and tool\-bound regimes, so equal hit counts need not hide equal amounts of time\. A predictor correct only on these long\-running calls would hide 97% of tool time by achieving only 17% hit rate\. Conversely, a predictor correct on all remaining calls would achieve an 83% hit rate but hide only 3%\. Thus, hit rate alone poorly reflects speculative latency savings\.

★\\bigstarInsight 1:Speculation value is concentrated: predicting a few slow calls can hide nearly all tool time\.

### 3\.2Environment Dependence Limits Observation Prediction

Observation speculation is limited by the information available in the actor’s context\. Figure[2](https://arxiv.org/html/2608.00881#S3.F2)\(c\) contrasts a predictable file\-comparison result with training metrics revealed only through execution\. More broadly, many latency\-critical observations depend on hidden environment state, such as files, running processes, or available resources, and therefore cannot be predicted reliably from the actor’s context alone\.

For observations beyond this information boundary, executing a predicted action early can supply the missing result, but only conditionally\. Since a mispredicted action may corrupt the environment, speculative execution can safely complement observation prediction only when contained in a sandbox that supports snapshotting and forking\.

★\\bigstarInsight 2:Environment\-dependent outcomes demand execution, not prediction: sandboxed action speculation reveals them early without risking committed state\.

### 3\.3Longer Lookahead Creates Runway but Collapses Prediction Accuracy

A speculative action launched when its issuing decode begins can hide at mostmin⁡\(Di,Ti\)\\min\(D\_\{i\},T\_\{i\}\)\. As decoding accelerates, this runway shrinks: Figure[2](https://arxiv.org/html/2608.00881#S3.F2)\(d\) shows that the one\-step overlap ceiling falls from 43\.3% to 25\.7% as per\-token latency decreases from 20 ms to 1 ms\.

Looking farther ahead restores the lost runway: at 1 ms, increasing the horizon from one to five steps raises the idealized ceiling from 25\.7% to 65\.2%\. However, prior multi\-step approaches require the entire predicted action chain to match\(Sui et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib21); Song[2026](https://arxiv.org/html/2608.00881#bib.bib20); Ye et al\.[2025](https://arxiv.org/html/2608.00881#bib.bib28)\), causing accuracy to collapse multiplicatively from 20\.9% at one step to 2\.4% at three and 1\.2% at five\. The speculative timing that offers the most runway is therefore the least likely to realize it\.

★\\bigstarInsight 3:Farther lookahead creates runway, but multiplicative chain errors erase the overlap it is meant to unlock\.

Together, these findings identify three barriers to useful speculation and motivate the design ofAOSpec\.

![Refer to caption](https://arxiv.org/html/2608.00881v1/x6.png)Figure 3:Overview ofAOSpecon a representative agentic trajectory\. \(a\) Serial execution exposes tool latency\. \(b\) Observation only speculation advances predictable outputs but cannot infer environment dependent results\. \(c\) Action only speculation offers little same step overlap and fragile exact chain lookahead\. \(d\)AOSpecdrafts observation with EVD and combines them with isolated target action lookahead, verifying action forks with JASV\.

## 4Methodology

### 4\.1Algorithm Overview

AOSpecaddresses the three requirements identified in Section[3](https://arxiv.org/html/2608.00881#S3): prioritize latency over hit rate, combine predicted and environment\-produced observations, and extend action overlap without predicting an entire future trajectory\. Figure[3](https://arxiv.org/html/2608.00881#S3.F3)illustrates howAOSpecresolves the identified challenges\.

At each verified observation boundary, the actor begins generating next action while speculative actions are launched in isolated sandbox forks\. Speculative candidates may target the current or a later window\. When the actor emits the target actionaa,*Joint Action–State Verification*\(JASV\) selects a fork only if both the speculative action matches the target action and the original environment matches the committed environment; otherwise,aaexecutes from the committed environment\.

A completed action supplies its actual observation immediately\. While execution remains pending,*Expected Value Decoding*\(EVD\) drafts candidate observations, each of which may start a provisional actor continuation\. When the actual observation returns, only the continuation whose draft is identical to the actual observation is kept\. The resulting environment is then committed, and invalid forks are pruned\. Algorithm[1](https://arxiv.org/html/2608.00881#alg1)summarizes the control flow ofAOSpec\.

Algorithm 1AOSpec: Action–Observation Co\-Speculation1:

H,S←H,S\\leftarrowverified history and committed environment

2:

B←StartActor​\(H\)B\\leftarrow\\textsc\{StartActor\}\(H\)
3:

ℱ←LaunchActions​\(H,S\)\\mathcal\{F\}\\leftarrow\\textsc\{LaunchActions\}\(H,S\)
4:whiletruedo

5:

x←FinishActor​\(B\)x\\leftarrow\\textsc\{FinishActor\}\(B\)
6:if

xxis a terminal responsethen

7:return

xx
8:endif

9:

a←xa\\leftarrow x
10:

f←f\\leftarrowa fork in

ℱ\\mathcal\{F\}satisfying

VerifyAction​\(f,a,S\)\\textsc\{VerifyAction\}\(f,a,S\), if one exists

11:ifno such

ffexiststhen

12:

f←Execute​\(S,a\)f\\leftarrow\\textsc\{Execute\}\(S,a\)
13:endif

14:

𝒫←∅\\mathcal\{P\}\\leftarrow\\varnothing
15:if

ffhas not finishedthen

16:Start

PredictObservations​\(H,a\)\\textsc\{PredictObservations\}\(H,a\)
17:foreach candidate

o^\\hat\{o\}that finishes before

ffdo

18:

𝒫​\[o^\]←StartActor​\(H∘\(a,o^\)\)\\mathcal\{P\}\[\\hat\{o\}\]\\leftarrow\\textsc\{StartActor\}\(H\\circ\(a,\\hat\{o\}\)\)
19:endfor

20:endif

21:

\(o,S\+\)←Wait​\(f\)\(o,S^\{\+\}\)\\leftarrow\\textsc\{Wait\}\(f\)
22:if

o∈dom⁡\(𝒫\)o\\in\\operatorname\{dom\}\(\\mathcal\{P\}\)then

23:

B′←𝒫​\[o\]B^\{\\prime\}\\leftarrow\\mathcal\{P\}\[o\]
24:else

25:

B′←StartActor​\(H∘\(a,o\)\)B^\{\\prime\}\\leftarrow\\textsc\{StartActor\}\(H\\circ\(a,o\)\)
26:endif

27:Cancel unfinished drafts and discard all provisional branches except

B′B^\{\\prime\}
28:

H←H∘\(a,o\)H\\leftarrow H\\circ\(a,o\);

S←Commit​\(S\+\)S\\leftarrow\\textsc\{Commit\}\(S^\{\+\}\)
29:

ℱ←PruneInvalid​\(ℱ∖\{f\},S\)∪LaunchActions​\(H,S\)\\mathcal\{F\}\\leftarrow\\textsc\{PruneInvalid\}\(\\mathcal\{F\}\\setminus\\\{f\\\},S\)\\cup\\textsc\{LaunchActions\}\(H,S\)
30:

B←B′B\\leftarrow B^\{\\prime\}
31:endwhile

### 4\.2Speculative Observation

While action execution remains pending, observation speculation advances actor generation\. Because a small fraction of calls contains most tool time \(Section[3\.1](https://arxiv.org/html/2608.00881#S3.SS1)\), it targets expected time hidden rather than hit rate\.

#### Optional Observation\-Model Fine\-Tuning

Observation\-model training could improve accuracy at smaller model sizes and candidate widths, reducing the compute required for drafting\. Given verified historyHtH\_\{t\}, actionata\_\{t\}, and observationoto\_\{t\}, we fine\-tune a small model by minimizing

ℒobs​\(θ\)=−∑\(Ht,at,ot\)∈𝒟log⁡pθ​\(o∣Ht,at\)\.\\mathcal\{L\}\_\{\\mathrm\{obs\}\}\(\\theta\)=\-\\sum\_\{\(H\_\{t\},a\_\{t\},o\_\{t\}\)\\in\\mathcal\{D\}\}\\log p\_\{\\theta\}\(o\\mid H\_\{t\},a\_\{t\}\)\.\(1\)We oversample costly windows to retain rare, latency\-critical outcomes\. Execution traces provide supervision directly, without manual labels or observation classes\.

#### Expected Value Decoding

EVD separates observation probability from tool time\. Given\(Ht,at\)\(H\_\{t\},a\_\{t\}\), it constructs a candidate set𝒞​\(Ht,at\)\\mathcal\{C\}\(H\_\{t\},a\_\{t\}\)using one or more proposal mechanisms and estimates each candidate’s associated tool time from historical executions\{\(oj,Tj\)\}\\\{\(o\_\{j\},T\_\{j\}\)\\\}:

T^​\(c\)=∑jK​\(c,oj\)​Tj∑jK​\(c,oj\),\\widehat\{T\}\(c\)=\\frac\{\\sum\_\{j\}K\(c,o\_\{j\}\)\\,T\_\{j\}\}\{\\sum\_\{j\}K\(c,o\_\{j\}\)\},\(2\)whereKKmeasures outcome similarity\. Candidates are ranked by

Vo​\(c\)=pθ​\(c∣Ht,at\)​T^​\(c\),V\_\{o\}\(c\)=p\_\{\\theta\}\(c\\mid H\_\{t\},a\_\{t\}\)\\,\\widehat\{T\}\(c\),\(3\)a proxy for expected tool time hidden\. The highest\-scoring candidates are drafted subject to the branch budget\.KKmay compare discrete outcome forms or learned representations; it affects ranking but not verification\.

Observation decoding races action execution\. Unfinished drafts are cancelled if execution finishes first; a draft that finishes first starts provisional target\-model generation for stept\+1t\+1\. EVD selects what to draft, while the race determines the realized saving\.

#### Observation Verification

When execution returnsoto\_\{t\},AOSpecretains only a continuation whose predicted observation is byte\-identical tooto\_\{t\}\. Mismatching continuations and their descendants are discarded; if none matches, generation restarts fromoto\_\{t\}\. Canonicalizing inconsequential fields such as process UUIDs could improve acceptance without changing application\-level semantics\.

### 4\.3Speculative Action

Observation prediction cannot recover outcomes determined by the execution environment \(Section[3\.2](https://arxiv.org/html/2608.00881#S3.SS2)\)\. Speculative action instead executes speculative, latency\-critical actions in isolated forks, obtaining their actual observations while containing their effects\. JASV verifies each speculative action and its original environment before reuse\.

#### Joint Action–State Verification

Each forkffrecords its speculative actiona^f\\hat\{a\}\_\{f\}and the versionν​\(Sf\)\\nu\(S\_\{f\}\)of its original environment\. When the actor emits the targetaia\_\{i\}from the committed environmentSiS\_\{i\}, JASV accepts the fork only if both actions and pre\-execution environments match:

ValidAct​\(f,ai,Si\)=\[a^f=ai\]∧\[ν​\(Sf\)=ν​\(Si\)\]\.\\mathrm\{ValidAct\}\(f,a\_\{i\},S\_\{i\}\)=\[\\hat\{a\}\_\{f\}=a\_\{i\}\]\\land\[\\nu\(S\_\{f\}\)=\\nu\(S\_\{i\}\)\]\.\(4\)Action equality covers the tool and its arguments\. A valid fork executed the same action from the same environment, so its observation and effects can be reused; otherwise, it is discarded andaia\_\{i\}executes fromSiS\_\{i\}\.

JASV replaces explicit dependency analysis by discarding outstanding speculative forks upon commit\. By performing speculative execution within the isolated fork,AOSpec’s execution is semantically identical to the serial execution, making it a lossless speculation\. In Algorithm[1](https://arxiv.org/html/2608.00881#alg1),PruneInvalidremoves stale forks after each commit, andVerifyActionchecks the action and version before reuse\.

#### State\-Verified Action Lookahead

Longer lookahead increases overlap, but conventionally requires a longer chain of correct predictions, whose acceptance probability decreases multiplicatively with the lookahead depth \(Section[3\.3](https://arxiv.org/html/2608.00881#S3.SS3)\)\. JASV instead predicts only the latency\-critical target action and later verifies its launch environment\.

If an action issued at stepiiis launched at boundaryjj, its available runway is

Rj,i=∑k=jiDk\+∑k=ji−1Tk,R\_\{j,i\}=\\sum\_\{k=j\}^\{i\}D\_\{k\}\+\\sum\_\{k=j\}^\{i\-1\}T\_\{k\},\(5\)allowing it to hide up tomin⁡\(Ti,Rj,i\)\\min\(T\_\{i\},R\_\{j,i\}\)of its execution time\.

A candidate may launch at any verified boundary and remains valid across intervening calls that leave the managed environment unchanged\. Thus, these calls need not be predicted, although forecasting a more distant target may remain harder\. Calls apparent from task context may also launch before the first actor step, similar to prefetching before task execution\.

#### Runtime Support

The sandboxed runtime provides snapshot, fork, execute, commit, and discard operations\. Our implementation assigns each sandbox’s copy\-on\-write \(CoW\) filesystem image an immutable version, which is uniquely identified by its root hash\. Forks record their origin version, and commit the verified speculative path, reducing JASV to a version comparison rather than a recursive scan\. Non\-filesystem inputs and effects are excluded\. Our implementation adopts a CoW agent runtime from concurrent work under anonymous review; we will add its public citation when available\.

## 5Experiments

![Refer to caption](https://arxiv.org/html/2608.00881v1/x7.png)Figure 4:End\-to\-end latency savings across nine Terminal\-Bench harness and actor configurations and five actor TPOTs\. Solid curves invoke an LLM for speculation; dashed curves do not\.![Refer to caption](https://arxiv.org/html/2608.00881v1/x8.png)Figure 5:SWE\-bench transfer with the observation model trained on Terminal\-Bench: \(a\) end\-to\-end latency savings and \(b\) total tool time hidden across actor TPOTs\.![Refer to caption](https://arxiv.org/html/2608.00881v1/x9.png)Figure 6:Expected\-value decoding over MSA×\\timesSonnet 4\.5 and Judy×\\timesOpus 4\.6 \(width 1, 10 ms TPOT\): \(a\) hidden tool time, \(b\) exact\-match accuracy, and \(c\) exact\-match hit rate are averaged across cells; \(d\) compares mined\- and embedding\-based value estimates for each cell\.![Refer to caption](https://arxiv.org/html/2608.00881v1/x10.png)Figure 7:Component and observation\-model ablations\. \(a\) Mean latency savings from observation speculation, action speculation, and their combination across nine Terminal\-Bench cells\. \(b\) Observation accuracy for fine\-tuned and frozen 0\.6B models on the same MSA×\\timesSonnet trials\.![Refer to caption](https://arxiv.org/html/2608.00881v1/x11.png)Figure 8:Action\-model ablations across nine Terminal\-Bench cells: trace\-matched action accuracy versus \(a\) candidate width and \(b\) lookahead distance\. Curves show the nine\-cell means; bands show the interquartile ranges\.### 5\.1Experimental Setup

##### Benchmarks and metrics\.

We evaluateAOSpecthrough trace replay on Terminal\-Bench 2\.0\(Merrill et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib15)\)and SWE\-bench Verified\(Jimenez et al\.[2024](https://arxiv.org/html/2608.00881#bib.bib12)\), spanning four agent harnesses—mini\-swe\-agent \(MSA\)\(Yang et al\.[2024](https://arxiv.org/html/2608.00881#bib.bib26)\), OpenHands\(Wang et al\.[2025](https://arxiv.org/html/2608.00881#bib.bib24)\), Gemini CLI\(Google[2025](https://arxiv.org/html/2608.00881#bib.bib10)\), and Judy111Judy is an anonymous submission listed on the Terminal\-Bench 2\.0 leaderboard\.—and five actor models and nine harness–model configurations on Terminal\-Bench 2\.0, plus MSA on SWE\-bench Verified with traces from the official “Bash Only” leaderboard\. We split Terminal\-Bench tasks into disjoint training and test sets, ensuring that every task category is represented in the test set\. Predictors use only training tasks\. Because the actors are accessed through APIs as black boxes and all optimizations operate within the harness, live scheduling could cause methods to follow different trajectories\. We therefore follow the community convention to replay all methods on identical actor trajectories\(Xia et al\.[2026](https://arxiv.org/html/2608.00881#bib.bib25)\), enabling reproducible comparisons across serving regimes\. We compute actor latency from recorded output token counts and sweep TPOT over\{20,15,10,5,1\}\\\{20,15,10,5,1\\\}ms/token, spanning conventional to low\-latency serving regimes\(Artificial Analysis[2026a](https://arxiv.org/html/2608.00881#bib.bib2),[b](https://arxiv.org/html/2608.00881#bib.bib3); Groq[2024](https://arxiv.org/html/2608.00881#bib.bib11); Cerebras Systems[2025](https://arxiv.org/html/2608.00881#bib.bib7)\)\. We primarily report total latency reduction relative to serial execution, alongside tool time hidden and exact prediction rate\.

##### Baselines\.

We compare against Speculative Actions \(*Spec\-Actions*\), which uses Qwen3\.6\-35B\-A3B\-FP8\(Qwen Team[2026](https://arxiv.org/html/2608.00881#bib.bib18)\)to predict and launch the next action\(Ye et al\.[2025](https://arxiv.org/html/2608.00881#bib.bib28)\), and two Speculative Observation baselines, the SpecHop variants\(Saberi, Rezaei, and Feizi[2026](https://arxiv.org/html/2608.00881#bib.bib19)\):*SpecHop\-cache*reuses prior observations, while*SpecHop\-LLM*uses Qwen3\.6\-35B\-A3B\-FP8 to draft observations\.*Oracle\-action*receives the correct next action from the trace and defines the one\-step action speculation ceiling\. Serial execution is the vanilla baseline\.

##### Hardware\.

Drafter generation runs on 2 NVIDIA H200 GPUs\. Tool calls are run on a Microsoft Azure Standard\_D4s\_v5 VM\. The VM has four vCPUs and 16 GiB of memory with an Intel Xeon Platinum 8370C \(Ice Lake\) processor at 2\.80 GHz\. Storage consists of a 64 GB Premium SSD OS disk and a 256 GB P15 Premium SSD data disk\.

##### Implementation\.

The observation drafter is Qwen3\-0\.6B, fine\-tuned for each harness\. The action drafter is Qwen3\.6\-35B\-A3B\-FP8 without additional training\. vLLM\(Kwon et al\.[2023](https://arxiv.org/html/2608.00881#bib.bib13)\)is used as the serving backend for drafting\. Unless stated otherwise,AOSpecuses five observation branches and eight action forks\.

### 5\.2End\-to\-End Performance and Tail Latency

##### Comparison with baselines\.

As shown in Figure[4](https://arxiv.org/html/2608.00881#S5.F4),AOSpecoutperforms every practical baseline across all 45 configuration and TPOT combinations\. Its mean saving across equally weighted configurations rises from 11\.8% at 20 ms TPOT to 32\.5% at 1 ms, compared with 4\.3% and 12\.2% for the strongest baseline\. Spec\-Actions provides negligible savings in most settings because long tool executions overlap only with much shorter actor decodes\. SpecHop\-cache performs well when observations recur, particularly with Gemini 2\.5 Pro under MSA and OpenHands and in the Judy configurations, but offers little benefit for Gemini CLI and most GPT\-5 mini settings, where costly outputs are less reusable\. In contrast,AOSpecremains effective across all configurations\. It also exceeds the one step action oracle in 41 of 45 settings because observation speculation and cross step action lookahead extend beyond the one step action only ceiling\. Its advantage grows as faster decoding exposes more tool time while reducing same step overlap\.

##### Tail latency\.

Table[1](https://arxiv.org/html/2608.00881#S5.T1)reports trial\-level latency at the representative 10 ms TPOT\.AOSpecachieves the lowest latency at every reported quantile\. It reduces median and p90 latency by 10\.7% and 10\.5%, while the reductions grow to 30\.3% at p95 and 42\.8% at p99\. The larger tail gains agree with the latency concentration identified in Section[3\.1](https://arxiv.org/html/2608.00881#S3.SS1)\.

Table 1:End\-to\-end latency distribution \(seconds\) per trial at 10 ms actor TPOT\.

### 5\.3Generalization to an Unseen Benchmark

Figure[5](https://arxiv.org/html/2608.00881#S5.F5)evaluates SWE\-bench using the observation model trained only on Terminal\-Bench traces\. At 1 ms TPOT,AOSpecachieves 18\.9% latency savings, compared with 4\.2% for the strongest baseline, demonstrating transfer to an unseen benchmark without retraining\.

### 5\.4Expected\-Value Decoding Targets Costly Calls

##### Effect of latency weighting\.

Figure[6](https://arxiv.org/html/2608.00881#S5.F6)\(a–b\) compares three decoding rules over candidates from the same observation model: estimated tool time, model likelihood, and their EVD product\. Tool time alone favors costly but implausible outcomes, whereas likelihood favors probable outcomes regardless of their latency\. By balancing both, EVD hides 60\.5 seconds per trial, versus 47\.6 seconds for likelihood and 39\.4 seconds for tool time\. It redirects correct predictions toward costly calls, raising the hit rate above 60 seconds from 18\.4% to 21\.0% while slightly reducing it for subsecond calls \(Figure[6](https://arxiv.org/html/2608.00881#S5.F6)\(c\)\)\. Thus, EVD hides more time without requiring more correct predictions\.

##### Similarity function\.

Figure[6](https://arxiv.org/html/2608.00881#S5.F6)\(d\) compares two implementations ofKK:*Pattern\-EVD*groups observations matched by the same mined regular expression pattern, whereas*Embedding\-EVD*groups observations that are nearby in a learned representation space\. Pattern\-EVD hides 40\.9 versus 22\.3 seconds per trial on MSA×\\timesSonnet 4\.5 and 80\.1 versus 61\.2 seconds on Judy×\\timesOpus 4\.6\. We therefore use Pattern\-EVD in the main evaluation\.

### 5\.5Ablation Studies

##### Two\-level speculation\.

Figure[7](https://arxiv.org/html/2608.00881#S5.F7)\(a\) shows that, at 10 ms TPOT, observation and action speculation alone save 12\.0% and 7\.1% on average, respectively, while their combination saves 17\.0%\. At 1 ms TPOT, the joint savings reach 32\.5%\. The two levels are complementary: observation speculation advances generation past predictable observations, while action speculation executes tools early to resolve environment\-dependent outputs\.

##### Observation\-model training\.

On the same MSA×\\timesSonnet trials, fine\-tuning the 0\.6B observation model raises top\-1 accuracy from 2\.4% to 29\.1% \(Figure[7](https://arxiv.org/html/2608.00881#S5.F7)\(b\)\)\. As candidate width increases to five, however, the frozen model reaches 28\.1% versus 32\.9% for the fine\-tuned model, narrowing the gap from 26\.7 to 4\.8 percentage points\. This convergence suggests that, with sufficient compute for multiple attempts,AOSpeccan operate training\-free using an off\-the\-shelf observation model\.

##### Action\-speculator accuracy\.

Increasing candidate width from 1 to 32 raises mean trace\-matched accuracy from 11\.0% to 23\.7%, trading additional forks for greater coverage \(Figure[8](https://arxiv.org/html/2608.00881#S5.F8.1)\(a\)\)\. At width 32, target\-action accuracy declines from 23\.7% at distance one to 7\.7% at distance ten \(Figure[8](https://arxiv.org/html/2608.00881#S5.F8.1)\(b\)\)\. By verifying the target action and its launch state directly, JASV keeps these long\-range predictions usable without requiring the intervening action chain to match, whose accuracy is effectively zero at this distance\.

### 5\.6Observation\-Model Context

Table 2:Observation model input ablation on MSA×\\timesSonnet 4\.5\. Each variant removes the indicated component during both training and inference; history contains all prior reasoning, tool calls, and observations\.##### Effect of input components\.

As shown in Table[2](https://arxiv.org/html/2608.00881#S5.T2), removing the task query or reasoning text does not decrease both hit rate and tool time hidden\. Removing prior observations or the entire interaction history lowers exact hit rate from 33\.3% to 30\.1% and 29\.9%, respectively, while hidden time remains stable\. We therefore condition the observation model on the current action and prior interaction history\.

## 6Conclusion

AOSpechides the latency of environment\-execution in agentic serving by jointly speculating observations and actions\. EVD targets latency\-critical observations by their expected time savings, while isolated execution and JASV safely obtain environment\-dependent outputs and enable long\-range action speculation without predicting entire action chains\. On representative agentic workloads,AOSpecreduces end\-to\-end latency by 11\.8–32\.5% and p99 latency by up to 42\.8%\. These results demonstrate that co\-speculating model outputs and stateful environment execution is an effective approach to low\-latency agent serving\.

## References

- Anthropic \(2026\)Anthropic\. 2026\.How Claude Code Works\.https://code\.claude\.com/docs/en/how\-claude\-code\-works\. Accessed July 24, 2026\.
- Artificial Analysis \(2026a\)Artificial Analysis\. 2026a\.Anthropic: Models Intelligence, Performance & Price Analysis\.https://artificialanalysis\.ai/providers/anthropic\.
- Artificial Analysis \(2026b\)Artificial Analysis\. 2026b\.OpenAI: Models Intelligence, Performance & Price Analysis\.https://artificialanalysis\.ai/providers/openai\.
- Bai et al\. \(2026\)Bai, H\.; Lv, W\.; Zheng, H\.; Lu, Y\.; and Shu, J\. 2026\.SPORK: Self\-Speculative Forking to Accelerate Agentic LLM Inference\.*arXiv preprint arXiv:2607\.03333*\.
- Cai et al\. \(2024\)Cai, T\.; Li, Y\.; Geng, Z\.; Peng, H\.; Lee, J\. D\.; Chen, D\.; and Dao, T\. 2024\.Medusa: Simple llm inference acceleration framework with multiple decoding heads\.*arXiv preprint arXiv:2401\.10774*\.
- Cao et al\. \(2026\)Cao, S\.; Chang, Z\.; Li, C\.; Li, H\.; Fu, L\.; and Tang, J\. 2026\.The Auton Agentic AI Framework\.*arXiv preprint arXiv:2602\.23720*\.
- Cerebras Systems \(2025\)Cerebras Systems\. 2025\.Cerebras Sets New AI Speed Record on MBZUAI and G42’s K2 Think at 2,000 Tokens/Second Inference Performance\.https://www\.cerebras\.ai/press\-release/k2\-think\-pr\.
- Chen et al\. \(2024\)Chen, H\. M\.; Luk, W\.; Yiu, K\. F\. C\.; Li, R\.; Mishchenko, K\.; Venieris, S\. I\.; and Fan, H\. 2024\.Hardware\-aware parallel prompt decoding for memory\-efficient acceleration of llm inference\.*arXiv preprint arXiv:2405\.18628*\.
- Dong et al\. \(2026\)Dong, Y\.; He, J\.; Liu, S\.; Hou, Y\.; Du, D\.; Xu, Z\.; Yu, S\.; Yang, B\.; Xia, Y\.; and Chen, H\. 2026\.DeltaBox: Scaling Stateful AI Agents with Millisecond\-Level Sandbox Checkpoint/Rollback\.*arXiv preprint arXiv:2605\.22781*\.
- Google \(2025\)Google\. 2025\.Gemini CLI\.https://github\.com/google\-gemini/gemini\-cli\.
- Groq \(2024\)Groq\. 2024\.Groq First Generation 14nm Chip Just Got a 6x Speed Boost: Introducing Llama 3\.1 70B Speculative Decoding on GroqCloud\.https://groq\.com/blog/groq\-first\-generation\-14nm\-chip\-just\-got\-a\-6x\-speed\-boost\-introducing\-llama\-3\-1\-70b\-speculative\-decoding\-on\-groqcloud\.
- Jimenez et al\. \(2024\)Jimenez, C\. E\.; Yang, J\.; Wettig, A\.; Yao, S\.; Pei, K\.; Press, O\.; and Narasimhan, K\. R\. 2024\.SWE\-bench: Can Language Models Resolve Real\-world Github Issues?In*The Twelfth International Conference on Learning Representations*\.
- Kwon et al\. \(2023\)Kwon, W\.; Li, Z\.; Zhuang, S\.; Sheng, Y\.; Zheng, L\.; Yu, C\. H\.; Gonzalez, J\.; Zhang, H\.; and Stoica, I\. 2023\.Efficient memory management for large language model serving with pagedattention\.In*Proceedings of the 29th symposium on operating systems principles*, 611–626\.
- Li et al\. \(2026\)Li, Y\.; Wei, F\.; Zhang, C\.; and Zhang, H\. 2026\.Eagle\-3: Scaling up inference acceleration of large language models via training\-time test\.*Advances in Neural Information Processing Systems*, 38: 136737–136756\.
- Merrill et al\. \(2026\)Merrill, M\. A\.; Shaw, A\. G\.; Carlini, N\.; Li, B\.; Raj, H\.; Bercovich, I\.; Shi, L\.; Shin, J\. Y\.; Walshe, T\.; Buchanan, E\. K\.; et al\. 2026\.Terminal\-bench: Benchmarking agents on hard, realistic tasks in command line interfaces\.*arXiv preprint arXiv:2601\.11868*\.
- Nichols et al\. \(2025\)Nichols, D\.; Singhania, P\.; Jekel, C\.; Bhatele, A\.; and Menon, H\. 2025\.Optimizing Agentic Language Model Inference via Speculative Tool Calls\.*arXiv preprint arXiv:2512\.15834*\.
- OpenClaw Foundation \(2026\)OpenClaw Foundation\. 2026\.OpenClaw Agent Runtime Architecture\.https://docs\.openclaw\.ai/agent\-runtime\-architecture\. Accessed July 24, 2026\.
- Qwen Team \(2026\)Qwen Team\. 2026\.Qwen3\.6\-35B\-A3B: Agentic Coding Power, Now Open to All\.https://qwen\.ai/blog?id=qwen3\.6\-35b\-a3b\.
- Saberi, Rezaei, and Feizi \(2026\)Saberi, M\.; Rezaei, K\.; and Feizi, S\. 2026\.SpecHop: Continuous Speculation for Accelerating Multi\-Hop Retrieval Agents\.*arXiv preprint arXiv:2605\.21965*\.
- Song \(2026\)Song, Y\. 2026\.B\-PASTE: Beam\-Aware Pattern\-Guided Speculative Execution for Resource\-Constrained LLM Agents\.*arXiv preprint arXiv:2604\.16469*\.
- Sui et al\. \(2026\)Sui, Y\.; Zhao, H\.; Ma, R\.; He, Z\.; Wang, H\.; Li, J\.; Xu, K\.; Chen, K\.; and Yang, Y\. 2026\.Parallelizing Tool Execution and LLM Generation for Low\-Latency Agent Serving\.*arXiv preprint arXiv:2603\.18897*\.
- Tencent Cloud \(2026\)Tencent Cloud\. 2026\.CubeSandbox: Instant, Concurrent, Secure & Lightweight Sandbox Service for AI Agents\.https://github\.com/tencentcloud/CubeSandbox\.
- Tile\-AI \(2026\)Tile\-AI\. 2026\.TileRT: Tile\-Based Runtime for Ultra\-Low\-Latency LLM Inference\.https://github\.com/tile\-ai/TileRT\.
- Wang et al\. \(2025\)Wang, X\.; Li, B\.; Song, Y\.; Xu, F\. F\.; Tang, X\.; Zhuge, M\.; Pan, J\.; Song, Y\.; Li, B\.; Singh, J\.; et al\. 2025\.Openhands: An open platform for ai software developers as generalist agents\.In*International Conference on Learning Representations*, volume 2025, 65882–65919\.
- Xia et al\. \(2026\)Xia, T\.; Li, H\.; Li, Z\.; Chen, X\.; Kang, H\.; Qiao, Y\.; Xu, Y\.; and Stoica, I\. 2026\.Idleness is Relative: Exploiting Tool\-Call Idle Windows for Offloading in Agentic Systems with MORI\.*arXiv preprint arXiv:2606\.00866*\.
- Yang et al\. \(2024\)Yang, J\.; Jimenez, C\. E\.; Wettig, A\.; Lieret, K\.; Yao, S\.; Narasimhan, K\. R\.; and Press, O\. 2024\.SWE\-agent: Agent\-Computer Interfaces Enable Automated Software Engineering\.In*The Thirty\-eighth Annual Conference on Neural Information Processing Systems*\.
- Yao et al\. \(2022\)Yao, S\.; Zhao, J\.; Yu, D\.; Shafran, I\.; Narasimhan, K\. R\.; and Cao, Y\. 2022\.React: Synergizing reasoning and acting in language models\.In*NeurIPS 2022 Foundation Models for Decision Making Workshop*\.
- Ye et al\. \(2025\)Ye, N\.; Ahuja, A\.; Liargkovas, G\.; Lu, Y\.; Kaffes, K\.; and Peng, T\. 2025\.Speculative Actions: A Lossless Framework for Faster Agentic Systems\.*arXiv preprint arXiv:2510\.04371*\.
- Zhong et al\. \(2026\)Zhong, S\.; Lu, B\.; Chen, Q\.; Liu, C\.; Yang, F\.; and Li, M\. 2026\.DualSpec: Accelerating Deep Research Agents via Dual\-Process Action Speculation\.*arXiv preprint arXiv:2603\.07416*\.