Tag
RAPTOR is a role-aware framework for differentially private fine-tuning of Mixture-of-Experts models, addressing failure modes like clipping interference and noise dilution, with experimental improvements over standard DP baselines.
This repository provides GGUF quantizations of the Qwen3.8-Flash-Next model using gradient-based methods GSQ and RCO for optimized low-bit representation, enabling efficient deployment in standard tools.
A llama.cpp implementation that expands MoE expert routing beyond native top-K during inference using adaptive thresholds and layer-specific linear decay, without requiring model retraining or fine-tuning.
The M-series models use sparse Mixture of Experts architecture, with M2 routing to 10B of 230B parameters per forward pass and M3 to 23B of 428B, allowing deployment on 4xH100 GPUs at specific pricing.
ToMoE proposes a method to convert dense large language models into Mixture-of-Experts models using dynamic structural pruning without weight updates, outperforming existing techniques.
A discussion on the merits of Mixture of Experts models with 124B total parameters but only ~5B active, suggesting that low-active MoEs may be the optimal local sweet spot.
This paper introduces an automated LLM pipeline to generate and verify human-readable descriptions of when individual weights in transformers matter, finding that 12-31% of weights in weight-sparse transformers are interpretable globally, outperforming dense transformers.
The author argues that there is an urgent need for AI models in the 80-160B parameter range to support users with unified memory devices (e.g., high-RAM Apple/AMD systems), as recent models are either too small or too large for their hardware.
DOT-MoE formulates dense layer decomposition as a differentiable optimal transport problem, enabling efficient training of sparse MoE models that retain 90% of original performance while reducing active parameters by 50%.
EMO is a Mixture-of-Experts model that enables modular deployment by grouping similar domain tokens with shared experts, achieving performance comparable to standard MoEs while allowing significant expert pruning (25% experts retain 99% performance) without performance degradation.
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.