augmented-llm

AugServe introduces a state-aware request scheduling framework with dynamic batch-level token budgets to mitigate head-of-line blocking and improve effective throughput for augmented LLM inference serving, achieving up to 6.5x higher throughput than vLLM.