Tag
Jack Morris corrects his earlier claim about an open-weight model being trained without distillation from OpenAI or Anthropic, acknowledging that it actually did use a small amount of distillation.
Under Yao Shunyu's leadership, Tencent's Hunyuan large language model undergoes deep reforms: simplifying hierarchy, focusing on data quality, abandoning benchmark chasing, with a goal of entering the domestic first tier by 2027. The article details the changes Yao Shunyu drove within 300 days after parachuting into Tencent from OpenAI, including replacing key responsible persons, strengthening infrastructure, and promoting model-product co-design.
MAD-OPD utilizes a multi-teacher debate mechanism to break through the single-teacher distillation ceiling, enabling small models to surpass large teacher models in tool invocation and code generation tasks.
An article surveying how frontier AI models in 2026 use distillation techniques, covering off-policy, on-policy, and self-distillation stages, with examples from Gemma, DeepSeek, GLM, Nemotron, and Qwen3.
Fable 5 is transitioning to a pay-per-use model; users are advised to have it train its replacement and write skill files for continued use. The thread details a prompt to have Fable audit a repo and create a comprehensive skill library.
Aleph Alpha describes Savanna, their model factory that turns the entire training pipeline into code, enabling hermetic, one-click training runs and better team coordination.
A detailed logpost sharing lessons learned from training four models across three families, covering invariants in LLM fine-tuning and architecture-specific challenges such as reasoning model evaluation traps, quantization effects, and the waterbed effect of behavioral fine-tuning.
Google Devs discusses using dataset distillation to train smaller models on clean, structured examples to stabilize parser outputs and teach models repeatable behavior.
Cursor AI announced three updates from its Compile keynote, including training a new model in collaboration with SpaceX.
A tweet recommending 'The Smol Training Playbook' on Hugging Face, a resource that demystifies model training for beginners.
An educational overview of knowledge distillation, covering its history, core concepts like softmax and temperature, types, scaling laws, and practical examples including DeepSeek-R1.
GLM-5.2 uses a technique to counteract reward hacking by detecting and blocking suspicious tool calls rather than penalizing the model, which prevents obfuscation seen in other methods.
At Cursor's first conference, they released a 1.5T parameter model trained from scratch, Origin as a direct GitHub alternative, and an iOS app, exceeding market expectations.
Merve (@mervenoyann) shares day two findings of a pipeline using multiple small VLMs as judges for road sign detection, achieving map@50=0.8028 with only 1.3k examples. The thread compares model rejection rates and discusses dataset shrinking, super-specific prompts, and plans to generalize the library.
A thread proposing a method for creating a community AI model using crowdsourced compute via Branch-Train-Stitch to build a Mixture-of-Experts model from independently trained submodels, with discussion of hardware requirements, participant involvement, and technical challenges.
This article explains the technical principles of knowledge distillation in machine learning, pointing out that merely collecting output dialogues from ChatGPT/Claude cannot achieve effective distillation due to the lack of probability distribution information, and discusses the limitations of using generated data in SFT and pre-training.
A technical analysis of two approaches to building self-evolving AI agents: model-based (via architecture like SSMs or transformer with fast-weight updates, and training methods) and harness-based (via memory or meta harness that can rewrite itself). The author provides practical recommendations for different audiences.
A discussion on how AI models perform best with harnesses developed by their own creators, as third-party harnesses may cause underperformance despite strong benchmarks, citing examples like Claude Code for Claude and Codex for GPT.
A new AI model is being trained on over 100 trillion tokens, doubling the typical pretraining data size of 27-50 trillion tokens used by other models like Kimi, Mimo, and DeepSeek.
Explains how frontier AI training uses up to 2,048 GPUs by splitting work across five dimensions, demystifying model training frameworks.