Tag
This paper introduces memory-augmented speculative execution for LLM agents, using three online memory systems to improve prediction accuracy by 19-39% on action prediction and up to 2.5x on observation prediction, all while being lossless with zero added wall-clock cost.
GPUHedge is an open-source tool that uses speculative execution to hedge between serverless GPU providers, reducing cold start p95 latency from 117s to 30s.
SpecHop is a continuous speculation framework that accelerates multi-hop retrieval agents by maintaining multiple speculative threads and verifying predictions asynchronously, achieving up to 40% latency reduction without changing final model outputs.
Accio is a speculative execution framework that reduces cost and latency for web agents by leveraging offline site-structure profiling and online selection of fast paths, achieving a 1.9x reduction in per-task cost and 33.4% latency reduction while maintaining accuracy.