@elliotarledge: For those wondering why I use a Kimi Linear megakernel instead of Qwen 3.6, first look at the parameter counts. One is …

For those wondering why I use a Kimi Linear megakernel instead of Qwen 3.6, first look at the parameter counts. One is 35 billion, one is 48 billion, and they’re both 3 billion active experts. So they’re going to use the same amount of weights in total for, or roughly the same amount of weights for predicting a single token, but the difference is in the number of total parameters.

Now notice how one of them has 27 layers and the other has 40. When we have a layer, that usually means we’re going to have more synchronizations overall. So synchronization means normalization, softmax, or one where you have to traverse the entire sequence to get a global row number and then to apply that on the row again or maybe to do multiple of those.

Those are issues which significantly bottleneck the speed of a single token prediction due to waiting for threads to line up at a barrier before contining with the next layer. So if you limit the number of layers, then your effort on the kernel goes more into how much performance you can get out of a single layer while having less synchronizations to do total, because there are less layers in total.

If there’s a higher hidden dimension, that means there’s more compute to do. Experts do not change. If you have double the MoE intermediate length from the up projection, you have more compute to put into that. If you’re focused on multi-head latent attention, you have to be cleverer inside the attention layers because you are compressing the model’s memory, whereas group query detention is very static and well-known already.

So being able to fuse an interesting attention type with a megakernel was a kind of taste decision that I made. For the linear type, I mean here it doesn’t matter that much, but I found that for specifically a Chinese lab like Kimi Delta Attention or KDA, if the Kimi lab saturates their models on that architecture, then we know why it performs well under a certain benchmark if Kimi architecture specific kernels performs well, but other megakernels do not.

So that gives us some intel into if labs are focusing on Chinese invented architectures or not. And then the vocab size means that we can worry less about the total number of kernels because if I have two different kernels where one is focused on the massive 250,000 vocab size versus just one kernel and we reduce the vocab size by 50%, making it more feasible to deal with it in the same kernel launch, that is much more feasible from a megakernel perspective.

So on almost every axis, the chimney linear architecture makes more sense to use as a mega kernel as opposed to the commonly known Qwen3.6 architecture that people will run on their RTX 3090s for their local agent workflows.

33% not 50% my bad

thats why i made it lol