TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
Summary
TAO 提出一个工具感知的联合调度框架，用于基于 LLM 的智能体服务，通过动态规划在容量约束下选择和放置会话，相比最优基线最高降低 26.8% 延迟、36.3% TTFT 并提升 29.5% 解码吞吐。
View Cached Full Text
Cached at: 10/29/26, 05:31 AM
Paper page - TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
Source: https://huggingface.co/papers/2610.37230
Abstract
TAO improves scheduling for LLM agent services by considering tool readiness and using dynamic programming to place sessions, reducing latency and increasing throughput.
Agentic services alternate between model inference on clustered GPUs and external tools on distributed CPUs. In this scenario, existing systems (1) overlooktool-return readinessin service evaluation, wasting limited resources, and (2) use heuristic strategies to decideadmission and placementon GPUs, resulting in suboptimal scheduling decisions. To bridge this gap, we present TAO, a tool-aware joint scheduling framework forLLM-based agentic serving. Its value interface combines near-term tool readiness,node-specific cache creditwithtime-aware decay, andtoken footprint; a dynamic program jointly selects and places waiting sessions under capacity constraints. Across the 12 reported model-workload-concurrency settings, TAO reduces latency by up to 26.8%, reduces meantime to first token (TTFT)by up to 36.3%, and increasesdecode throughputby up to 29.5% relative to the best evaluated state-of-the-art baseline for each metric.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.37230 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2610.37230 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.37230 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Competence-Aware Orchestration for LLM-Based Multi-Agent Systems
该论文提出了一种两阶段的 competence-aware 框架，用于 LLM 多智能体系统的自动编排，根据模型在当前查询上的能力动态调整角色激活与通信图，在六个基准上平均达到 93.81%，同时在高能力骨干上降低 36.26% 的 token 成本。
TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving
TOPAS is a workflow-aware prefix-state scheduling method for multi-agent LLM serving that optimizes cache usage and request scheduling to reduce job completion time.
Think in Strategies, Act with Tools: Traceable Agentic Optimization for Combinatorial Problems
提出 Tool-Augmented Strategic Agent (TASA)，通过总结和利用优化轨迹、结合策略引导的自适应工具规划，提升 LLM 基于提示的组合优化（OPRO）的可追溯性与解质量。
@mylifcc: Major Experiment: Using LLM as an 'Optimization Agent' for Automatic Loop Scheduling! Just read this paper accepted at PACT 2025: 'Agentic Auto-Scheduling: An Experimental Study of LLM-Guided Loop Opt…'
Introduces a paper accepted at PACT 2025, proposing the ComPilot framework, which uses off-the-shelf LLMs as optimization agents to automatically optimize complex loop nests without fine-tuning, achieving a geometric mean speedup of 3.54x, surpassing the SOTA Pluto.
AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving
AugServe introduces a state-aware request scheduling framework with dynamic batch-level token budgets to mitigate head-of-line blocking and improve effective throughput for augmented LLM inference serving, achieving up to 6.5x higher throughput than vLLM.