@no_stp_on_snek: Dense vs MOE - Can MOE keep up with dense quality? spawned from a problem a friend is having. he has an agentic run tha…
Summary
A comparison of dense and Mixture of Experts (MoE) models to assess if MoE can match dense model quality in agentic tasks, based on benchmark tests with various models showing performance metrics.
View Cached Full Text
Cached at: 08/26/26, 03:32 AM
Dense vs MOE - Can MOE keep up with dense quality?
spawned from a problem a friend is having. he has an agentic run that takes 5 hours using gemma4. it’s a bit slow for him. he’s trying to decide whether or not to get a spark to speed him up but can’t sacrifice on quality. so can an MOE model with fewer active parameters actually keep up?
so I took his independent tests and ran them against a bunch of models, moe and dense. Ornith 1.5 was the first one I compared that was close so focused a lot on that
it’s close enough to start testing in his pipeline but not sure if the end result will actually produce equal quality to gemma4 because there is evident variance in the failures which begs the question, how many of the pass results also have variance?
so should he use you use the moe over dense? it depends on your use case. answering a subset of questions to measure “quality” won’t be enough. he’ll have to do full runs and compare outputs over many iterations.
After i tested ornith i tested a bunch of other models. as you can see, the trusty old qwen3-Code-Next (and qwen lineup) did quite well:
full 3,700-question runs results Gemma 4 31B Turbo3 3,447–3,459 | 93.16–93.49%
Qwen3-Coder-Next Q8 3,444 | 93.08%
Qwen3.8-27B FP8 3,442 | 93.03%
Gemma 4 Turbo4 3,437 | 92.89%
Ornith 1.5 35B-A3B Q8 3,413 | 92.24%
Nemotron 3 Super 120B-A12B NVFP4 3,407 | 92.08%
Ornith abliterated NVFP4 3,324 | 89.84%
DeepSeek V4 Flash abliterated Headroom128 1,958 | 52.92%
note: the questions are a private curated list that should have judgeable answers that are either correct or not.
Similar Articles
What is the point of MoE models, beyond being faster?
A discussion about the advantages of Mixture of Experts (MoE) models over dense models beyond speed, considering RAM constraints and scaling limits.
Dense vs. MoE gap is shrinking fast with the 3.6-27B release
A new 3.6-27B release shows MoE closing the performance gap with dense models, especially in coding tasks and large context windows, though dense still leads overall.
MobileMoE: Scaling On-Device Mixture of Experts
MobileMoE introduces efficient on-device mixture-of-experts language models with sub-billion parameters, achieving better performance and efficiency than dense baselines and existing MoE models. The models are trained on open-source datasets and demonstrate significant speedups on commodity smartphones.
Mixture of Experts (MoEs) in Transformers
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.
I measured it: swapping a dense 27B for a 30B-A3B MoE changes the local concurrency ceiling (same rig as prior test, one variable change)
The author tests and compares the concurrency performance of dense versus MoE AI models on a MacBook Pro, finding that MoE models scale significantly better due to lower memory bandwidth usage per token.