Tag
The paper proposes EDAR, an entropy-driven adaptive routing framework that decides at inference time whether to answer a query with retrieval-augmented generation or escalate it to full long-context processing. EDAR retains 97.4% of the pure long-context baseline's accuracy while cutting token cost by 70.7%, escalating only 18.2% of queries on LongBench v2 and Infinity-Bench.
This paper presents a hybrid amortized inference method that accelerates hierarchical sparse predictive coding by combining a fast initial estimate with corrective refinement steps, achieving better efficiency than pure iterative or amortized approaches.
A user benchmarks thread count for hybrid CPU-GPU inference with Gemma 4 in llama.cpp, discovering a 80% performance uplift by using 16 threads instead of 6 on a hybrid core CPU, and shares the optimal command configuration.
Perplexity unveiled a hybrid local-cloud inference system at Computex 2026 that intelligently routes queries between on-device and cloud models, building on its earlier Personal Computer agent.
A developer built a routing layer on vLLM to route simple agent steps to a cheap open-source MoE model (21B active) and hard steps to Opus, reducing costs to $15.60 for a 400-step refactor with 93.4% success rate.