@bojie_li: Just finished watching Teacher He Jiyan from Zhongguancun College lead 7 PhD students to train a 7B model from scratch …
Summary
Teacher He Jiyan and 7 PhD students from Zhongguancun College trained a 7B model from scratch in 3 months, achieving state-of-the-art performance in the 7B model category, with insights on efficient training and data quality.
View Cached Full Text
Cached at: 09/26/26, 06:55 AM
Just finished watching Teacher He Jiyan from Zhongguancun College lead 7 PhD students to train a 7B model from scratch in 3 months, handling everything from pre-training to mid-training and post-training entirely on their own, achieving SOTA level in the 7B size model category.
First, although many people are hyping RSI, the technical report points out that the current model’s capabilities are far from fully autonomous, still only reaching L2-level assistance in areas like model architecture design, learning algorithm design, and data cleaning. I have the same feeling myself—no matter Astra or Fable, neither can replace my architecture design, and I have to keep reminding myself not to outsource my thinking.
Second, model training is a classic case of “easy for those who know how, impossible for those who don’t.” For those who know how, it doesn’t take much computational resources, but for those who don’t, no amount of compute power or headcount piling will get it done. For example, MiMo V2.6 RL only cost a little over 3 million USD, which is very cheap for a base model company. The entire MiMo core team has only a few dozen formal employees and didn’t take shortcuts like distillation. But models that burn over 100 million USD and invest thousands of people aren’t guaranteed to succeed. For Zhongguancun College, one teacher plus 7 students completing data creation, pre-training, mid-training, and post-training in just 3 months is extremely impressive.
Finally, data is the new code, and data quality is as crucially important as code quality. In the past few years, I’ve always tried to implement Agent self-evolution through code, but quickly hit a ceiling. Only in the last year did I realize that these things I was doing via code should instead be expressed through training data and baked into the model. We all know the harms of dirty code—dirty data is exactly the same. This 7B model from Zhongguancun College did extremely in-depth data work: pre-training data cleaning and curriculum learning, mid-training with context length expansion, post-training using open-source and distilled traces to build foundational capabilities like instruction following and long context, then constructing higher-order capabilities like long chain-of-thought reasoning and tool calling; in RL, continuously removing already high-probability solved problems to maintain GRPO learning efficiency.
Technical report:
Similar Articles
@mylifcc: They took 2.3 million pieces of 'thinking process' data left by Claude (a very powerful and expensive AI) during inference, and used it to train a very small model (Qwen3-4B, only 4 billion parameters). As a result, this small model performed very 'consistent' in tests: In 512 different tests, the output was exactly the same every time...
Using 2.3 million pieces of Claude inference trajectory data to distill the Qwen3-4B model, resulting in a small model with 100% output consistency and extremely low hallucination. Moreover, the student model was not limited by the teacher model, and it also converged to a universal truth.
@berryxia: Small model, big wisdom? It's now real! A 7B small model now acts as the boss of top large models like GPT-5, Claude Sonnet 4, Gemini 2.5 Pro. A new paper shows an RL-trained 7B model learned to write natural language subtasks, assign them to different models, precisely...
A new paper proposes training a 7B small model via reinforcement learning as a task scheduler, automatically decomposing subtasks and assigning them to top models like GPT-5 and Claude. It surpasses individual frontier models on several hard benchmarks, demonstrating that end-to-end reward learning can effectively replace manual prompt engineering and multi-agent pipeline design.
@zhixianio: After receiving the new machine, I began an 'ascetic' practice of forcing myself to use local models for common tasks. I thought it would be painful, but both speed and quality greatly exceeded my expectations: Model: Qwen3.6-35B-A3B-oQ6-fp16-mtp, Running: oMLX, with N…
The author uses the Qwen3.6-35B-A3B model and oMLX tool on the new local machine for daily tasks, finding that both speed and quality far exceed expectations, even outperforming remote LLMs in PA and coding scenarios, demonstrating a significant improvement in on-device AI capabilities.
@f14bertolotti: Stellar performance from a 3B model. These results were achieved primarily through post-training refinements on Qwen2.5…
This technical report introduces VibeThinker-3B, a 3B parameter model that achieves frontier-level verifiable reasoning performance through post-training refinements on Qwen2.5-Coder, including curriculum-based supervised fine-tuning, multi-domain reinforcement learning, and offline self-distillation, matching or exceeding much larger models like DeepSeek V3.2.
@alex_verem: a team of researchers just proved you don't need a bigger model, you need a smarter plan researchers from Tsinghua and …
Researchers from Tsinghua and South China University of Technology introduced Atomic Task Graph (ATG), a framework that enables 7B-8B open-source models to surpass GPT-4 on complex agent benchmarks without fine-tuning, by using directed graph-based planning and internal simulation to drastically reduce hallucination rates.