Tag
A user benchmarks thread count for hybrid CPU-GPU inference with Gemma 4 in llama.cpp, discovering a 80% performance uplift by using 16 threads instead of 6 on a hybrid core CPU, and shares the optimal command configuration.
Perplexity unveiled a hybrid local-cloud inference system at Computex 2026 that intelligently routes queries between on-device and cloud models, building on its earlier Personal Computer agent.
A developer built a routing layer on vLLM to route simple agent steps to a cheap open-source MoE model (21B active) and hard steps to Opus, reducing costs to $15.60 for a 400-step refactor with 93.4% success rate.