Tag
This paper presents an empirical study on scheduling multiple LLMs on shared heterogeneous hardware, focusing on performance implications of CPU-GPU offloading and preemption. It finds that offloading causes non-linear decode degradation, especially for smaller models, and preemption overhead is dominated by model state reload, providing design guidance for future multi-model schedulers.
A Kaggle competition challenges participants to build a scheduler that decides whether to run a 2B-parameter model on MMLU questions to minimize a weighted cost based on compute and accuracy trade-offs.