GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving
Summary
GORGO introduces a proxy architecture for LLM inference that jointly optimizes network latency, prefill cost, and queueing delay using evolutionary strategy tuning on a new synthetic dataset, improving p95 TTFT by 6.9-15.5% and end-to-end latency by 14.3-30.9%.
View Cached Full Text
Cached at: 07/07/26, 06:42 AM
Paper page - GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving
Source: https://huggingface.co/papers/2602.11688
Abstract
GORGO is a proxy architecture that optimizes LLM inference load balancing by jointly considering network latency, prefill cost, and queueing delay through evolutionary strategy tuning on a new synthetic dataset.
Increasingly,LLM inference servicesproxy client requests to engine replicas distributed globally.Load-balancing policiesmust jointly account for factors includingKV-cache locality,replica load, and variablenetwork latencywhen optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factorsnetwork latency,prefill cost, andqueueing delayusing tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M,evolutionary strategiesguide the GORGO policy’s parameters to directly optimizep95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improvep95 TTFTby 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baselineload-balancing policiessuch as simplesession affinityandprefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO.
View arXiv pageView PDFGitHub4Add to collection
Get this paper in your agent:
hf papers read 2602\.11688
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2602.11688 in a model README.md to link it from this page.
Datasets citing this paper1
#### alessiotoniolo/ART-Chat-2.5M Viewer• Updated4 days ago • 2.53M • 93
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2602.11688 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents
Memory-R2 introduces LoGo-GRPO, a training framework that combines local and global group-relative optimization to provide fairer credit assignment for long-horizon memory-augmented LLM agents, improving accuracy and inference latency across backbones.
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
This paper proposes a Mixture of LoRA and Full (MoLF) fine-tuning framework that uses gradient-guided optimizer routing to adaptively switch between LoRA and full fine-tuning. It aims to overcome the structural limitations of relying solely on static adaptation methods by combining the plasticity of full tuning with the regularization of LoRA.
Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads
A paper proposing a latency-aware LLM query router that jointly optimizes latency, accuracy, and cost using a lightweight latency estimator, achieving up to 40% improvement in accuracy–cost utility while maintaining comparable latencies.
Routing Without Training: Controllable-Ratio LLM Offloading via Reliability Gating
This paper introduces CARGO, a training-free routing framework that uses the local LLM's own inference-time agreement across sampled responses to decide when to offload to a cloud model, enabling controllable collaboration ratios without additional training.
Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking
This paper presents a method to fine-tune LLaMA 3 8B as an efficient reranker for Retrieval-Augmented Generation using knowledge distillation and 4-bit quantization, achieving 14-21% gains in retrieval metrics over cross-encoder baselines with reduced inference cost.