head-of-line-blocking

AugServe introduces a state-aware request scheduling framework with dynamic batch-level token budgets to mitigate head-of-line blocking and improve effective throughput for augmented LLM inference serving, achieving up to 6.5x higher throughput than vLLM.

This paper presents a duration-aware scheduling approach for automatic speech recognition (ASR) serving that uses audio duration as a proxy for job time to reduce head-of-line blocking latency under workload drift.