Tag
This paper proposes a multi-objective optimization framework for routing in LLM serving, employing online linear programming with bid-price control to balance latency, throughput, and tail performance, and demonstrates improvements over heuristics using the Vidur simulator.