@liumengxinfly: Redis creator speaks out on X, saying that those who keep claiming Chinese models are distilled don't understand machine learning at all.

X AI KOLs Timeline News
knowledge-distillation machine-learning model-training technical-explanation chinese-ai controversy

Summary

This article explains the technical principles of knowledge distillation in machine learning, pointing out that merely collecting output dialogues from ChatGPT/Claude cannot achieve effective distillation due to the lack of probability distribution information, and discusses the limitations of using generated data in SFT and pre-training.

I saw Redis creator speaking out on X, saying that those who keep claiming Chinese models are distilled don't understand machine learning at all. Earlier, when someone said that Chinese models were distilled, I would argue against it. But later, I've seen it so many times that I'm tired of it and just decided to write it out. Many people talk about distillation without actually knowing what distillation is. What they might mean is that collecting a large amount of Claude/ChatGPT conversations can make their own models as capable as Claude/ChatGPT, but they never consider the technical feasibility. Take an image recognition example: if there is a model that can recognize animals, and I want to distill it into my own model. If I simply give the student model images and the teacher model's classification results, the distillation is meaningless. Because this is no different from training directly with labeled samples—in fact, the labels might be more accurate than the teacher model's results, and using the teacher model's results could even reduce accuracy. If the teacher model's classification results are meaningless for distillation, then what is meaningful? What is meaningful is the weights of each category in the layer before the final result. For example, if the classification result is "cat", the previous layer might have probabilities: cat 80%, tiger 5%, leopard 7%, dog 0%... This layer actually represents the model's predicted probability distribution, which is what the student model should learn. Because this contains more implicit information than the final single result—for instance, cats are more similar to leopards and less similar to dogs—which is closer to the real-world probability distribution. In LLMs, it's similar. It seems that an LLM generates the next token, but actually it generates a probability distribution for the next token. Essentially, the parameters of an LLM are fixed, so for the same input, the output is also fixed. The reason each conversation generation is different is that, during output, the model does not choose the token with the highest probability; instead, it randomly picks a token within a certain probability range, resulting in output randomness. This is what parameters like Temperature and Top-k do: adjust the final probability distribution to increase output randomness. Therefore, simply collecting a large number of Claude/ChatGPT conversation outputs cannot achieve true distillation, because you cannot obtain the probability distribution information, and thus cannot make the model converge quickly to resemble Claude/ChatGPT. What about using these conversation outputs for SFT? After the concept of distillation expanded, some people also call this type "corpus distillation." Theoretically, it is possible, but SFT is typically used to align language style and preferences. It is generally believed that SFT sacrifices the generalization ability and intelligence level of the large model, making it more like a specialized model. I personally lean towards the view that this approach cannot improve the model's capabilities. Another possibility is to directly include these conversation contents as part of the pre-training corpus. In existing model papers, we can see that pre-training already uses generated data. However, it is generally believed that capabilities like tool invocation and agent abilities need to be elicited through large-scale RL. I am skeptical about how much these capabilities can be activated during pre-training. Moreover, pre-training actually requires more general abilities, and you cannot let a large amount of generated data contaminate the overall data quality. I personally guess that a portion of pre-training data comes from Claude/ChatGPT—it might even be hard to avoid, since they have already become part of the world's data—but relying on this alone is far from achieving similar coding and agent capabilities. You see, others just casually say "model distillation," but I have to go through so much trouble to explain. I really don't want to talk about it anymore.
Original Article
View Cached Full Text

Cached at: 06/17/26, 03:46 AM

I saw a post from Redis’s author on X, saying that people who keep insisting Chinese models are just distilled don’t understand machine learning at all.

I used to argue back when people said Chinese models are distilled. But after seeing it so many times, I got tired and stopped bothering. So I decided to just write it down.

Many people throw around the term “distillation” without actually knowing what it means. What they really mean is that if you collect a huge amount of Claude/ChatGPT conversations, your model could somehow become as capable as Claude/ChatGPT — without ever considering the technical feasibility.

Take an image recognition example. Suppose there’s a model that can recognize animals, and I want to distill that knowledge into my own model. If I only feed the student model the input images along with the teacher model’s classification results, that distillation is meaningless. It’s the same as training with labeled samples — in fact, the labels you have might even be more accurate than the teacher’s outputs, so using the teacher’s results could actually reduce accuracy.

If the teacher’s classification results aren’t useful for distillation, then what is? What matters is the weight distribution from the second-to-last layer for each class. For example, if the classification result is “cat,” the previous layer might show probabilities like cat 80%, tiger 5%, leopard 7%, dog 0% … This layer is actually the model’s predicted probability distribution — that’s what the student model should learn. Because it contains much more implicit information than just the final single answer: cats are more similar to leopards than to dogs, which is closer to the true distribution of the real world.

It’s similar in LLMs. LLMs seem to just generate the next token, but they actually generate a probability distribution for the next token. Essentially, an LLM’s parameters are fixed, so with the same input, the output distribution is fixed too. The reason every conversation is different is that during output generation, you don’t always pick the token with the highest probability — you randomly choose one within a certain probability range, which introduces randomness. That’s what parameters like Temperature and Top-k do — they adjust the final probability distribution to increase output diversity.

So collecting vast amounts of Claude/ChatGPT conversation outputs alone cannot achieve strict distillation. Because you can’t obtain the probability distribution information, you can’t make your model converge quickly to what Claude/ChatGPT looks like.

Then what about using these conversation outputs for SFT? After the concept of distillation expanded, some call this “data distillation” as well. Theoretically it’s possible, but SFT is typically used to align language style and preference. It’s generally believed that SFT sacrifices the model’s generalization ability and intelligence, making it more like a specialized model. I personally tend to think this approach cannot improve the model’s core capabilities.

Another possibility is to directly include these conversation outputs as part of the pretraining corpus. Some recent model papers show they already use synthetic data during pretraining. But it’s generally believed that tool-calling and agent capabilities need to be activated through large-scale RL — I’m skeptical how much they could contribute at the pretraining stage. Moreover, pretraining requires more general capabilities; you can’t let too much synthetic data pollute overall data quality. My personal guess is that a small portion of pretraining data may come from Claude/ChatGPT — it’s almost unavoidable, since they’ve become a significant part of the world’s data. But relying on that alone is far from enough to achieve similar coding and agent capabilities.

See — someone casually says “model distillation,” and I have to explain it in such detail. I really don’t want to talk about it anymore.

Similar Articles

@0xcherry: https://x.com/0xcherry/status/2067610347633025281

X AI KOLs Timeline

This article analyzes the reasons behind the performance leap of Zhipu GLM-5.2, suggesting that its 40B activation parameters provide greater effective capacity after accounting for fixed overhead, making RL post-training more effective. It also reviews the history of Chinese AI model development and notes that the large model approach ultimately prevailed.

@Sxy_Cherotich: Recently I've been talking with quite a few model researchers, and a consensus conclusion is: the importance of data is once again highlighted. A while ago I got to know ex-Kimi's @FanqingMengAI, who is doing a startup in the data direction, and invited him to record a podcast. The biggest non-consensus from our conversation is Fanqing's view on the difference between domestic and foreign models...

X AI KOLs Timeline

A podcast about AI model competition, discussing the importance of data, distillation and pre-training innovation, and an interview with Evolvent AI co-founder Meng Fanqing, covering topics such as synthetic data, RSI, and differences in domestic models.

@snowboat84: Have you noticed that the birth of models in AI is actually quite arbitrary? Take language models as an example: first RNN, then LSTM, one day Transformer is said to be effective so everyone switches to it, later it's split into Encoder and Decoder, one moment BERT is all the rage, the next GPT is said to have emergent abilities and Scaling Law. The whole process hardly has any theoretical guidance.

X AI KOLs Timeline

The article discusses the arbitrariness of AI model creation, proposing to draw inspiration from physics models, build a repository of candidate models, and formalize the model selection process.

@vivilinsv: A serial entrepreneur I particularly like and admire, @quxiaoyin Xiaoyin, has been on fire on X recently. She made a very sharp judgment: Chinese open-source models will continue to gain market share, and may even become one of the "worst-case scenarios" for the US AI ecosystem—if Chinese models not only occupy the model layer...

X AI KOLs Timeline

Discusses how Chinese open-source models could become a threat to the US AI ecosystem, and the future competitive landscape between open-source and closed-source models, sparking widespread discussion in the AI community.

@Xx15573208: I've read many articles about Transformers and understand the theory, but when I actually sit down to write code, I have no idea where to start. LLMs-from-scratch is specifically designed to solve this problem: it accompanies the book "Build a Large Language Model" and guides you through implementing GPT from scratch using PyTorch…

X AI KOLs Timeline

LLMs-from-scratch is a GitHub repository that accompanies the book "Build a Large Language Model," providing complete code to implement GPT from scratch with PyTorch, covering the full pipeline including pretraining, fine-tuning, and RLHF. It has gained 93K+ stars and is ideal for developers who want to deeply understand the principles behind large language models.