GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving
Summary
GORGO introduces a proxy architecture for LLM inference that jointly optimizes network latency, prefill cost, and queueing delay using evolutionary strategy tuning on a new synthetic dataset, improving p95 TTFT by 6.9-15.5% and end-to-end latency by 14.3-30.9%.
View Cached Full Text
Cached at: 07/07/26, 06:42 AM
Paper page - GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving
Source: https://huggingface.co/papers/2602.11688
Abstract
GORGO is a proxy architecture that optimizes LLM inference load balancing by jointly considering network latency, prefill cost, and queueing delay through evolutionary strategy tuning on a new synthetic dataset.
Increasingly,LLM inference servicesproxy client requests to engine replicas distributed globally.Load-balancing policiesmust jointly account for factors includingKV-cache locality,replica load, and variablenetwork latencywhen optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factorsnetwork latency,prefill cost, andqueueing delayusing tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M,evolutionary strategiesguide the GORGO policy’s parameters to directly optimizep95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improvep95 TTFTby 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baselineload-balancing policiessuch as simplesession affinityandprefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO.
View arXiv pageView PDFGitHub4Add to collection
Get this paper in your agent:
hf papers read 2602\.11688
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2602.11688 in a model README.md to link it from this page.
Datasets citing this paper1
#### alessiotoniolo/ART-Chat-2.5M Viewer• Updated4 days ago • 2.53M • 93
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2602.11688 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents
Memory-R2 introduces LoGo-GRPO, a training framework that combines local and global group-relative optimization to provide fairer credit assignment for long-horizon memory-augmented LLM agents, improving accuracy and inference latency across backbones.
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
This paper proposes a Mixture of LoRA and Full (MoLF) fine-tuning framework that uses gradient-guided optimizer routing to adaptively switch between LoRA and full fine-tuning. It aims to overcome the structural limitations of relying solely on static adaptation methods by combining the plasticity of full tuning with the regularization of LoRA.
Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
This paper introduces GLoRA, a gauge-aware server representation for Federated LoRA that addresses the semantic mismatch in factor aggregation by estimating a consensus update subspace. Experiments show GLoRA outperforms baselines in performance and efficiency across heterogeneous client scenarios.
Local LLM Inference Optimization: The Complete Guide
A comprehensive guide to optimizing local LLM inference on consumer hardware, covering tools like llama.cpp, vLLM, and LM Studio, with practical advice on memory hierarchy, layer placement, and common failure modes.
LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling
This paper introduces AutoTTS, an environment-driven framework that automates the discovery of test-time scaling strategies for LLMs by formulating it as controller synthesis. It demonstrates improved accuracy-cost tradeoffs on mathematical reasoning benchmarks with minimal computational overhead.