Tag
This paper presents a tail-risk-aware scheduling method for agentic LLM workflows that reduces tail latency by optimizing turn release decisions, achieving up to a 3.50x speedup in P95 workflow flow time under contention.
This paper presents an empirical study on scheduling multiple LLMs on shared heterogeneous hardware, focusing on performance implications of CPU-GPU offloading and preemption. It finds that offloading causes non-linear decode degradation, especially for smaller models, and preemption overhead is dominated by model state reload, providing design guidance for future multi-model schedulers.
A Kaggle competition challenges participants to build a scheduler that decides whether to run a 2B-parameter model on MMLU questions to minimize a weighted cost based on compute and accuracy trade-offs.