TurboServe: Serving Streaming Video Generation Efficiently and Economically
Summary
TurboServe is a serving system designed for streaming video generation workloads, using an online scheduling approach that coordinates session placement and GPU provisioning to reduce latency and cost.
View Cached Full Text
Cached at: 07/02/26, 03:46 AM
Paper page - TurboServe: Serving Streaming Video Generation Efficiently and Economically
Source: https://huggingface.co/papers/2606.19271
Abstract
TurboServe is a specialized serving system for streaming video generation that addresses session state management and dynamic resource allocation challenges through integrated scheduling, autoscaling, and migration mechanisms.
Streaming video generationis emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving,streaming video generationmust preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods. We present TurboServe, the first serving system designed specifically forstreaming video generationworkloads. TurboServe formulates serving as anonline schedulingproblem that jointly coordinates session placement andGPU provisioning. Itsclosed-loop schedulingalgorithm combines amigration-aware placementcontroller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with aload-driven autoscalingcontroller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implementscoalesced chunk processingfor batching concurrent active sessions on the same GPU,GPU-CPU offloadingfor session suspension and resumption, andNCCL-based GPU-GPU migrationfor online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.
View arXiv pageView PDFProject pageGitHub10Add to collection
Get this paper in your agent:
hf papers read 2606\.19271
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.19271 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.19271 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.19271 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Streaming Video Generation with Streaming Force Control
StreamForce is a causal, unified video generation model that provides real-time, physically grounded responses to time-varying forces through a distillation pipeline and autoregressive architecture, achieving state-of-the-art performance in force adherence and motion realism.
Stream-T1: Test-Time Scaling for Streaming Video Generation
Stream-T1 is a proposed framework for test-time scaling in streaming video generation, improving temporal consistency and quality through mechanisms like noise propagation and reward pruning. The paper addresses the high computational costs of existing diffusion-based methods by leveraging chunk-level synthesis.
@songhan_mit: SANA Streaming: V2V on a single 5090
SANA Streaming enables video-to-video generation on a single NVIDIA RTX 5090 GPU.
@svpino: Serveless, but for models, is here! I stopped renting servers 12 years ago. I mostly moved everything to serverless fun…
Runpod announces FlashBoot, a serverless approach for AI models that moves idle models to cheaper storage and pages them back to GPU, achieving cold starts under 200ms and cutting costs by 90% compared to traditional clouds.
Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation
Stream-R1 introduces a reliability-perplexity aware reward distillation framework for streaming video generation that adaptively weights supervision to improve visual and motion quality without additional computational overhead.