moe

A developer successfully ran a 5.2 million parameter MoE LLM quantized to INT4 on an ESP32 Dev Kit V1 using only 81KB of SRAM by streaming experts from flash, achieving about 5 tokens per second.

Ahmad Osman argues that dense models like Qwen 27B perform poorly on unified-memory systems such as NVIDIA DGX Spark, suggesting MoE models are a better fit; he claims discrete GPUs like the RTX PRO 6000 deliver far better performance for agentic workloads.

Gregory Kurtzer praises Lucas Atkins' talk on lessons learned from training a large sparse Mixture-of-Experts model and life at an AI lab startup.

A developer benchmarks Qwen3.6-35B-A3B Q6 on an RTX 3090, showing that offloading eight MoE expert layers to CPU and increasing batch sizes improves prompt processing by 2.36× (564→1330 tok/s) with no decode speed regression, using evolutionary search to find the tuning config.

This paper introduces MESH, a memory-efficient Sinkhorn-based optimizer for Mixture-of-Experts (MoE) training that restores temporal momentum without storing full optimizer state, reducing memory by 62.5% while maintaining competitive evaluation loss compared to AdamW.

Mference, a Swift + Metal inference engine, now supports Inkling-Small 276B-A12B, running it at ~2.9 tok/s on under 10GB memory, enabling large MoE models on consumer Apple hardware.

jabbatheduck released a GGUF quantized version of the REAP expert-pruned DeepSeek-V4-Flash checkpoint, aggressively compressed for memory-constrained inference on consumer GPUs while preserving router and attention precision.

Cursor is open-sourcing Mixture-of-Kittens (MoK), a production MoE training megakernel for NVL72s that fuses communication and computation, delivering a 1.41x end-to-end training throughput improvement for their Composer model.

DeepGrove releases Maple-Preview, an open-source 20B-A1B ternary-weight reasoning LLM with SOTA reasoning for its weight class, capable of 200+ tokens/sec on a Mac mini M4 and competitive with larger models.

Maple-Preview is a ternary 20B MoE model that runs at 120 tokens per second on an iPhone, showcasing efficient on-device inference.

A llama.cpp PR adds heatmap-based caching of 'hot' MoE experts on the GPU, improving tok/s by ~1.7-2x for some models on 8GB VRAM, but with mixed results for others.

Su Jianlin reviews the K3 architecture, focusing on the combination of KDA + MLA + Stable LatentMoE + AttnRes. He explains the design trade-offs, MoE stability improvements, why MLA was kept, and the relationship between DSV4 and MLA.

A project claims to run Kimi K3, a 2.78T parameter MoE model, on CPU with 8GB RAM by streaming expert weights from NVMe SSD and using MXFP4 compression, trading speed for memory efficiency.

Discusses Apple's AFM3 20B model architecture, which uses Instruction-Following Pruning to activate only ~20% of MLP layers per prompt and keep the model in flash storage for efficient on-device inference.

A research paper presenting scaling laws and design principles for mixture-of-experts diffusion language models, trained as LLaDA MoE v2 (30B-A3B) on 23.5T tokens, approaching Qwen3 on several benchmarks with fewer pretraining tokens.

Frac-Connections is a new method that partitions hidden states instead of expanding them, extending Hyper-Connections to fractional expansion rates. It achieves stronger downstream performance than residual connections with virtually no extra compute, as validated on up to 7B MoE language models.

DeepSeek-V4-Flash-0731 is released in GGUF format at 2.88 bits per weight (95 GiB), making the 284B-parameter MoE runnable on 128 GB machines. Includes a detailed operating guide and a required TurboQuant llama.cpp fork.

A developer showcases Mference, a new inference engine that runs MoE models like DeepSeek-V4-Flash on just ~5.3GB of memory by streaming experts from SSD, with a native Mac app and OpenAI-compatible server.

This is a highly experimental GGUF version of the 2.8T-parameter Kimi K3 MoE model, with 55% of experts pruned and quantized to ~2.15 bpw (319 GiB). It requires a specific llama.cpp PR and custom patches to run, and includes detailed instructions for usage.

FedWeave proposes asymmetric aggregation for federated MoE-LoRA to handle task heterogeneity by separating expert aggregation from router optimization, achieving better specialization and performance.