@seclink: There are a few obvious interview questions: 1. Can Transformer do large-model inference? It can, so why not use it, and why must we use sglang and vllm? - Essentially because Transformer's performance is too poor; the bottlenecks are memory (KV cache) and slow token-by-token decoding...

There are a few obvious interview questions:

1. Can Transformers handle large model inference? Yes, they can, so why not use them directly, but instead have to use sglang, vllm?

   • Actually, at its core, it’s still because Transformer performance is just too underwhelming, with bottlenecks in VRAM (KV cache) and slow token-by-token decoding.
   • Going back to 2023, everyone was still tinkering with minor innovations on Transformers, like shared attention KV heads, FlashAttention using IO-awareness to save HBM reads/writes; on quantization and pruning, squeezing weights down to 4bit; on speculative decoding, small models guess, large models verify; architecturally, DeepseekMoE with sparse activation, pagedattention for paged KV management to eliminate storage fragmentation.

2. In 2024, quantization extends to KIVI/Qserve (compressing KV cache to 2-4bit, tailored for the context VRAM wall), FP8 natively bare on Hopper; enhancements bring multi-head decoding heads, ditching external draft models; on the scheduling side, RadixAttention becomes standard (prefix reuse tech), continuous batching and chunked prefill become standard.

3. 2025-2026, hardware-software integration. The paradigm shifts to making models inherently inference-friendly from the training stage. Model structures and GPU features are co-designed, attention evolving from MLA (Deepseek-v3’s breakthrough) to NSA (hierarchical sparse matrices); EAGLE-3 (feature-level drafts), MTP (self-speculation, even skipping draft [thinking process])

4. Quantization using NVFP4 (hardware-supported, performance boosted)

5. PD separation (costs drop another 30-40%)

Overall, large model evolution tech ties closely to actual work divisions in big companies—any problem can be tackled by categorizing, layering, and prioritizing. Some performance waste? Just iteratively refine it, and you’ll squeeze out the cost-performance ratio.

2. How has the architecture of KV Cache optimization evolved?

• Under standard MHA, KV grows linearly with sequence length x layers x heads, and at 128K context it starts exceeding weight VRAM. That’s a big problem.

• Architecture layer: many people try to shrink parameters, compress cache at the source, MQA -> GQA -> MLA -> NSA -> hybrid SSM route. Just make it not so damn sparse (currently in the industry, model activation is usually only 5%-7%, a lot of sparse matrix parameters are wasted).

• System layer - careful memory budgeting, pagedattention -> prefix/radix -> continuous batching -> chunked prefill -> offload/mooncake. Earlier we mentioned that under memory + sparsity there will be a lot of fragments, right? So treat VRAM as virtual memory paging. pagedattention solves fragmentation, then cross-request reuse (radixattention prefix tree, yes, the version of the red-black tree you’re familiar with), then optimize batching (continuous batching, reusing the same requests as much as possible) -> then balance prefill/decode (chunked) -> finally overflow cache to CPU/NVMe or split into two stages, from the ‘no waste’ principle to ‘reusable’ to ‘can overflow/decouple, can increase concurrency’.

• Token-level - patching afterwards. H20 -> streamingLLM -> snapKV -> pyramid/DuoAttn -> Ada-KV.

• Quantization. Precision compression, from FP8/int8 to KIVI 2bit to KVquant 3bit to GEAR/Atom.

• Retrieval/sparsity: quest/infllm -> Minference -> NSA native sparse, load on demand (yes, it’s the ‘lazy loading’ concept commonly used in frontend, some infrequently used things don’t need ‘preloading’).

• Merging: MiniCache / KVsharer -> Cam/D2O. Similar content doesn’t need to be stored multiple times in KV, only indexes.

3. How has large model distillation technology evolved? What does the latest look like now?

• 2015, one-hot was popular, the father of reinforcement learning said directly memorizing answers is too wasteful (BERT was also popular then, easy to overfit), and proposed that the essence of distillation is that large models should learn the thinking, not the answer. This is the ‘root cause’ of all technological development.

• 2016-2019, SeqKD proposed whole-sentence distillation, DistilBert proposed compression (retaining 97% capability).

• 2024, the LLM era exposed distribution mismatch (emergence). When distilling autoregressive large models, old methods collapsed. Students learn teacher outputs during training, but generate their own outputs during inference, so errors accumulate.

• 2024-2025, three adjustable knobs (Forward KL [safe but mediocre] -> Reverse KL (inverse reinforcement learning, sharp but prone to collapse) -> JSD -> Skew KL (DistiLLM adaptive), just tuning parameters, utterly useless, a year wasted on trial and error.

• 2024, engineering compression. Retraining is still expensive, first structurally prune, then continue training with distillation, reducing training cost. (Large models distill data, used as data source to retrain small models)

• 2025, paradigm shift, from distribution to reasoning process learning (learning of thinking’s thinking). This is the most powerful step. R1 generates 800k CoT reasoning chains, fed to small models (1.5B-80B) via SFT, and small models achieve o1-mini level reasoning at 1/20 the cost. The distillation target upgrades from output distribution (weights) to (learning your learning method [thinking process, which foreigners hype as meta-learning]), and concurrently SPIN proves even a classroom is unnecessary—self-play / left-right hand combat (self-play self-improvement).

• De-white-boxing: black-box / teacher-free emerges in parallel. White-box obtaining logits is limited by architecture, while the black-box route (synthetic data SFT, PAD preference-aligned distillation) and teacher-free (SPIN) free distillation from dependence on teacher internals, moving toward “strong models generate data, weak models learn behavior” (that is, learning QA, problem-solving thought processes, and the thinking that produces those thought processes together).

• 2015, one-hot was popular, the father of reinforcement learning said directly memorizing answers is too wasteful (BERT was also popular then, easy to overfit), and proposed that the essence of distillation is that large models should learn thinking, not answers. This is the ‘root cause’ of all technological development — at that time there was a bible in the RL field called “The Bitter Lesson”, which said don’t manually label, just throw everything before labeling to the large model and let it learn and reflect by itself.