Solved: LLM inference on Windows was 2–3x slower when the server window wasn't focused

Reddit r/LocalLLaMA News

Summary

LLM inference on Windows was 2–3x slower when the server window wasn't focused due to CPU scheduling delays; running the server detached/headless fixed the issue, maintaining consistent performance.

The fix: run the server detached/headless instead of keeping it attached to a console window. RTX 5090, ~27B NVFP4 model via ninfer: Terminal focused: 130–200 tok/s Terminal unfocused: 50–60 tok/s Click the terminal → immediately back to 130+ tok/s At first I thought GPU throttling, but the GPU wasn't the problem. During the slow runs: SM clock was actually higher: 2550–2600 MHz vs 2100–2200 MHz Power limit was the same: 400 W No PCIe power-state drop decode-host stayed around 390–494 µs The big difference was wait: wait tok/s Unfocused 38–41 ms 50–60 Focused 16–17 ms 130–220 So the GPU wasn't getting slower. The serving loop was just getting delayed on the CPU side. Running the server detached (docker run -d / headless) fixed it: wait stays around 17 ms and throughput stays around 130–200 tok/s, regardless of which window is focused. I also reproduced the same thing with a native Windows build, so this isn't WSL2-specific. Seems to be some kind of Windows foreground/background CPU scheduling behavior. Has anyone else seen this with local LLM inference on Windows 11?
Original Article

Similar Articles

WebLLM: high-performance in-browser LLM inference engine

Hacker News Top

WebLLM is a high-performance in-browser LLM inference engine that leverages WebGPU for hardware acceleration and is fully compatible with the OpenAI API, enabling local execution of open-source language models.

@_avichawla: This is the strangest property of LLM inference: Your LLM's output depends on requests that aren't yours. The same prom…

X AI KOLs Timeline

A technical explainer on why LLM outputs can vary between identical requests: floating-point accumulation order changes with batch size on the GPU, causing tied tokens to flip and completions to diverge even at temperature 0. It highlights batch-invariance flags in vLLM, SGLang, and DeepSeek's DeepInfer, noting the throughput trade-offs for reproducible outputs.