@AI_Whisper_X: The gains from LLM model architectures have already been exhausted. I read an article about model architectures: with so many changes to LLM architectures since 2020, which ones actually help when evaluated under the same experimental setup? The author, Nikita Pavlichenko (formerly worked on JetBrains Mellum pretraining…

X AI KOLs Timeline Papers
llm-architecture ablation-study scaling-laws mixture-of-experts linear-attention inference-cost model-design

Summary

Nikita Pavlichenko, former head of pretraining for JetBrains Mellum, systematically evaluated LLM architecture changes since 2020 through 1,129 training experiments. He concluded that architectural gains had largely been exhausted by the DeepSeek-V3 generation, with fine-grained MoE and linear attention representing the most substantial advances, while the impact of training methods such as Muon and inference kernels is often underestimated.

The gains from LLM architecture have already been exhausted I read an article about model architecture: of all the changes to LLM architectures since 2020, which ones actually help when tested under the same experimental setup? The author, Nikita Pavlichenko (former pretraining lead for JetBrains Mellum), ran 1,129 training experiments. He trained small models, with context lengths and quality evaluations limited to 16k; loss for the target large models was extrapolated using scaling laws. 1/ The question shouldn’t be “Is this architecture good?” What really matters is how good a model you can ultimately get for the same inference cost. He gives every architecture the same inference budget, so architectures that are cheaper to serve can afford larger models. With this yardstick, many conclusions flip: GQA, MQA, and sliding-window attention actually deliver worse quality per parameter, but shrinking the KV cache lets you use models 4–9 times larger, so they win overall. Conversely, if you compare intelligence per parameter alone, full MHA beats GQA, MQA, and sliding-window attention, while MLA matches it. More broadly, any architecture paper that reports only loss per parameter deserves scrutiny, because it leaves deployment costs out of the equation. 2/ The most solid advances over the past six years are fine-grained MoE and linear attention. MoE makes one active parameter do the work of 4–5 parameters in a dense model. It is the easiest win, and its advantage grows with scale. Hybrid linear-attention architectures (KDA, Gated DeltaNet, Mamba-2) improve quality while being 2–7 times cheaper. But most of linear attention’s advantage comes from properties it shares with SWA; in the complete 2026 model, it is only 0.3% better than sliding-window attention. He suspects this may partly explain why Western open-source labs are in no hurry to adopt linear attention. 3/ Most of the architectural gains had already been captured by the DeepSeek-V3 generation, leaving much less room for improvement afterward. He breaks down the loss: only about 2% of the predicted loss for the 2026 model comes from having too few parameters, and that is the portion architecture can improve. For GPT-3 in 2021, that figure was 11%. In other words, the DeepSeek-V3 generation had already captured most of that opportunity. This also helps explain why everyone’s architectures are converging, and why rumors that “frontier labs are training the most boring Transformers” sound credible. It isn’t laziness; there really isn’t much room left. 4/ Ablation findings on small models cannot be taken at face value: gains shrink by roughly a factor of 1.5–3, and some even reverse sign. This applies to everyone following nanoGPT speedruns and new modded-nanogpt records. His rule of thumb is practical: if a change helps by less than 1% at d=512, don’t add it without testing a second width. Multiplying embeddings by √d went from helping by 0.73% to hurting by 0.79%—a concrete example. 5/ Data and training methods may be more worthwhile investments than further architectural tweaking. He found that simply switching the optimizer to Muon could deliver gains comparable to a whole round of architectural changes combined. An ordinary architecture with a well-tuned training recipe could easily beat a top-tier architecture with an ordinary training recipe. And as scale increases, Muon’s gains hold up better than those from most small architectural tweaks. He didn’t test data changes, but said he expected gains of a similar magnitude. 6/ The extent to which inference kernels determine an architecture’s success is underestimated. Gated DeltaNet-2 performs best per parameter at the smallest width, d=512, with a measured −4.7%, but vLLM runs it at only 0.42 times KDA’s speed, leaving just −2.9% at equal serving cost. MLA also didn’t make the final model this time, partly because the models were too small and partly because vLLM is slow on small MLA shapes; he says MLA would very likely have made the cut if the target model had been a little larger. Architecture research and inference engineering are now inseparable. He also offered an assessment of AI doing research: it is already very good at implementation and debugging, but its research judgment is not yet reliable enough. AI can implement an architecture in half an hour that would take him a day, yet miss the most obvious and promising combinations to try. Still, he thinks it will soon catch up in this area too, just as it has in coding.
Original Article
View Cached Full Text

Cached at: 11/02/26, 04:15 PM

The gains from LLM architectures have already been exhausted

I read an article about model architectures: of all the changes to LLM architectures since 2020, which ones actually help when tested under the same experimental setup? The author, Nikita Pavlichenko (former head of pretraining for JetBrains Mellum), ran 1,129 training experiments. He trained small models, with context lengths and quality evaluations going only up to 16k; losses for the target large models were extrapolated using scaling laws.

1/ The question shouldn’t be “Is this architecture good?” What really matters is how good a model you can ultimately get at the same inference cost. He gave every architecture the same inference budget, so architectures that were cheaper to serve could use larger models. With this yardstick, many conclusions flip: GQA, MQA, and sliding-window attention actually deliver worse quality per parameter, but reducing the KV cache lets them support models 4–9 times larger, so they win overall. Conversely, if you compare intelligence per parameter alone, full MHA beats GQA, MQA, and sliding-window attention, while MLA matches it. More broadly, conclusions from architecture papers that report only loss per parameter deserve a question mark, because they leave deployment costs out of the equation.

2/ The most solid advances over the past six years are fine-grained MoE and linear attention. With MoE, one active parameter does the work of 4–5 parameters in a dense model. It’s the easiest win, and it gets better with scale. Hybrid linear-attention architectures (KDA, Gated DeltaNet, Mamba-2) improve quality while being 2–7 times cheaper. But most of linear attention’s advantage comes from what it shares with SWA; in the complete 2026 model, it is only 0.3% better than sliding-window attention. He suspects this may partly explain why Western open-source labs aren’t rushing to adopt linear attention.

3/ The architectural gains were largely captured by the DeepSeek-V3 generation, and the remaining room for improvement has clearly shrunk. He broke down the loss: in the predicted loss for the 2026 model, only about 2% comes from insufficient parameters, and that is the component architecture can improve. For GPT-3 in 2021, that room was 11%. In other words, the DeepSeek-V3 generation had already captured most of it. This also helps explain why everyone’s architectures are becoming increasingly similar, and why rumors that “frontier labs are training the most boring Transformers” sound credible. It isn’t laziness; there really isn’t much room left.

4/ Ablation findings from small models can’t be taken at face value: gains typically shrink by a factor of roughly 1.5–3, and some even reverse sign. This applies to everyone following nanoGPT speedruns and new modded-nanogpt records. His rule of thumb is practical: if a change helps by less than 1% at d=512, don’t add it without testing a second width. Multiplying embeddings by √d went from helping by 0.73% to hurting by 0.79%—a concrete example.

5/ Data and training methods may be more worthwhile investments than further architectural tinkering. He found that simply switching the optimizer to Muon could deliver gains comparable to an entire round of architectural changes combined. An ordinary architecture with a well-tuned training recipe could easily beat a top-tier architecture with an ordinary training recipe. Muon’s gains also hold up better as scale increases than those from most small architectural changes. He didn’t test data, but said he expected gains of a similar magnitude there.

6/ The extent to which inference kernels determine an architecture’s success is underestimated. Gated DeltaNet-2 was best per parameter at the smallest width, d=512, with a measured loss reduction of 4.7%. But in vLLM, it ran at only 0.42 times KDA’s speed, leaving just a 2.9% reduction at equal serving cost. MLA also didn’t make it into the final model this time, partly because the model was too small and partly because vLLM wasn’t fast on small MLA shapes. He said MLA would very likely have made the cut if the target model had been a little larger. Architecture research and inference engineering are now inseparable.

He also offered an assessment of AI’s research abilities: implementation and debugging are already strong, but research judgment is still unreliable. AI can implement an architecture in half an hour that would take him a day, yet miss the most obvious and promising combinations to try. Still, he thinks this gap will close soon too, just as it did with coding.

Nikita Pavlichenko (@nv_pavlichenko): Finally releasing the results of the arch ablations I was running with bux over the weekend

Similar Articles

@wangxiaoxin88: If you want to truly master LLM inference, this is worth studying closely. Someone put together an LLM Engineer's Practical Handbook — time-to-first-token. Half an hour a day, 10 weeks, 50 lessons total, building a complete production-ready tech stack from scratch: start with a mental model, understand why Decode blows up GPU memory and why Prefill saturates compute...

X AI KOLs Timeline

An open-source LLM engineer's practical handbook called time-to-first-token, offering a 10-week, 50-lesson roadmap to build a production-ready, OpenAI-compatible inference service from scratch — covering vLLM source code, PagedAttention, monitoring, 1000+ concurrency load testing, and quantization optimization.

@Phoenixyin13: Finished reading a long post today by OpenAI researcher Noam Brown — a reality severely underestimated by the industry. The true ceiling of LLM capabilities is far higher than what any current benchmark shows. The reason: too little test-time compute. And as models...

X AI KOLs Timeline

Highlights OpenAI researcher Noam Brown's argument: the true ceiling of LLM capabilities is far higher than current benchmarks show, due to insufficient test-time compute, and stronger models benefit more from additional computation. This poses a serious challenge for AI safety evaluation, as many dangerous capabilities may only emerge under long time and high compute budgets.

@Potatoloogs: How LLMs Actually Work Inside: From Token to Next-Token – A Complete Overview of Nine Core Mechanisms a) Tokenization: The model doesn't read text, it reads integers · Text is first split into subword pieces, then mapped to integer IDs; modern LLM vocabularies typically have tens of thousands to...

X AI KOLs Timeline

This article systematically outlines the nine core mechanisms inside modern LLMs, from tokenization to next-token prediction, including tokenization, embedding, positional encoding, attention, multi-head attention, feed-forward networks, etc., and compares architectural differences between various models.