Tag
This paper introduces Speculative Macro Commit (SMC), a two-tier agent system that reduces latency in tool-using LLM agents by speculatively executing future action chains and committing them when matched, achieving speedups over sequential execution.
The article proposes speculative programmatic tool calling (sPTC), a technique to optimize tool calls in AI harnesses by overlapping execution with token generation, reducing latency inspired by speculative execution in CPUs and LLMs.
Introducing Speculative Programmatic Tool Calling (sPTC), a technique to speculate on tool calls during code generation to overlap with token generation and execution time, improving efficiency in AI harnesses.
Uber open-sourced SubmitQueue, a high-performance speculative merge queue that validates multiple changes in parallel against predicted future states of HEAD to keep trunk green at scale.
A paper introducing predictive speculative KV replication to handle bursty LLM inference workloads, with code available on GitHub.
This paper introduces memory-augmented speculative execution for LLM agents, using three online memory systems to improve prediction accuracy by 19-39% on action prediction and up to 2.5x on observation prediction, all while being lossless with zero added wall-clock cost.
GPUHedge is an open-source tool that uses speculative execution to hedge between serverless GPU providers, reducing cold start p95 latency from 117s to 30s.
SpecHop is a continuous speculation framework that accelerates multi-hop retrieval agents by maintaining multiple speculative threads and verifying predictions asynchronously, achieving up to 40% latency reduction without changing final model outputs.
Accio is a speculative execution framework that reduces cost and latency for web agents by leveraging offline site-structure profiling and online selection of fast paths, achieving a 1.9x reduction in per-task cost and 33.4% latency reduction while maintaining accuracy.