Proposed architecture for inferencing sparse MOE models increasing Active parameters using layered + linear decay. Succinct reasoning without any model training or fine tune. [p]
Summary
A llama.cpp implementation that expands MoE expert routing beyond native top-K during inference using adaptive thresholds and layer-specific linear decay, without requiring model retraining or fine-tuning.
Similar Articles
Multi Tier MoE Caching
Discusses multi-tier caching strategies for MoE models to improve inference speed by keeping frequently activated experts on GPU, referencing existing implementations like PowerInfer and llama.cpp branches.
Increasing active parameters per token in MOE (Qwen 35B A4B+) reduce reasoning token by 8.5% - and you don't need to train or finetune!
A paper introduces an inference-time optimization for sparse MoE models by adjusting expert selection in late transformer layers, reducing reasoning tokens by 8.5% and latency by 10.9% without retraining, while maintaining accuracy.
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
FlexMoE proposes a one-for-all nested intra-expert pruning method for MoE language models, enabling multiple deployable subnetworks from a single training run with minimal performance loss.
Sticky Routing: Training MoE Models for Memory-Efficient Inference
StickyMoE proposes a differentiable routing consistency loss that encourages adjacent tokens to activate the same experts in MoE models, reducing expert-swapping overhead and cache misses during inference on edge devices by up to 3.92× while improving perplexity.
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
MoE2-LoRA introduces a dual-channel Routing-Conditioned Projection and a global LoRA expert pool to enable MoE-style low-rank adaptation for fine-tuning MoE models, achieving state-of-the-art accuracy while retaining general capabilities.