workload-management

AOS-Bench, from Tsinghua University and Columbia researchers, evaluates LLM coding agents as workload managers handling streams of tasks under shared CPU and memory limits, finding that isolated task success doesn't carry over to streaming scenarios and that simple fixed-concurrency policies often match elaborate agent-managed ones.

A paper proposing a latency-aware LLM query router that jointly optimizes latency, accuracy, and cost using a lightweight latency estimator, achieving up to 40% improvement in accuracy–cost utility while maintaining comparable latencies.