real-numbers

#real-numbers

tried predicting which MoE experts get used next token to speed up cpu/gpu offload, got some real numbers, is this actually implementable or am i wasting my time (30tg/s -> 150-200tg/s)

Reddit r/LocalLLaMA ↗ · 2026-07-16

Explores predicting which MoE experts will be used for the next token to improve CPU/GPU offloading, achieving 30->150-200 tg/s speedup, and questions the feasibility of implementation.

0 favorites 0 likes

real-numbers

tried predicting which MoE experts get used next token to speed up cpu/gpu offload, got some real numbers, is this actually implementable or am i wasting my time (30tg/s -> 150-200tg/s)

Submit Feedback