model-free-control

Tag

Cards List
#model-free-control

Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits

arXiv cs.AI ↗ · 2026-10-01 Cached

The paper introduces Decode-Latency Feedback Prefill (DLFP), a model-free proportional controller implemented in vLLM that adapts chunked prefill sizing based on observed decode latency, reducing P99 inter-token latency by ~28% on Qwen3-0.6B on an A100, while also documenting a clear generalization failure on larger models and multi-GPU setups.

0 favorites 0 likes
← Back to home

Submit Feedback