@superalesha: I took Qwen3.8-27B apart to see how it works inside. The plan was to carve a MoE out of it. Every ffn neuron tapped, al…

I took Qwen3.8-27B apart to see how it works inside. The plan was to carve a MoE out of it.

Every ffn neuron tapped, all 1,114,112 of them. 3.92M tokens pushed through. 87 minutes on the 4x3090.

The result surprised me. Neurons that fire on more than 90% of tokens: 2.

Thats what youre looking at, one frame per token. Full numbers below

The recipe from the papers starts by gluing the always-on neurons into a shared expert. On this model that population is two neurons wide.

One smooth tail, no split to cut along.

The quiet ones are real though: 19.6% of neurons almost never fire, and together they carry 2.9% of all firing.

There is slack, its just not where you would put it.

Last layer: 87% of its neurons almost never fire. Middle of the model, layers 16 to 31: only 9%.

So cutting every layer by the same amount is the wrong move. You undercut the tail and you butcher the middle, and the middle is doing the work.

Qwen3.8-27B alternates 48 cheap linear attention layers with 16 full attention ones. I assumed the ffn sitting above each type would pick up different habits.

It doesnt. Medians 0.00229 against 0.00223. Same shape, same everything.

Null result, but its a new one.

The top-64 neurons I count as fired carry only 5 to 14% of the layers actual activation mass. Rank by that and you keep the soloists while you mute the choir.

Perfect router at 50% budget: keeps 99.9% of the fired neurons, but only 57% of the energy. Picking 54% of neurons at random keeps 54%.

So the clever selection barely beats a coin flip on the thing that probably matters.

The experts are carved. Stage 3 is the eval that settles it.

My bet: 50% survives ugly, 25% is dead on arrival, and layer 63 with its 3 experts breaks first.

Yeah, I need to finish it up and then I’ll share all the source code

Bro, this is just the beginning. The chances of success are minimal

It’s not clear yet. The benchmark will be