training

Introduces 'Training Under Challenge', an executable-certificate framework that uses architecture-valid procedures to construct alternative models and estimate the empirical global-optimality gap of neural network checkpoints, with theoretical guarantees and experiments on ResNet-18 distillation and quantized denoising.

Andrej Karpathy built an LLM from scratch and open-sourced all code, covering the full pipeline of dataset, training loop, and inference loop, continuing his work on simplifying AI education with micrograd and makemore.

The author discusses a paper that demystifies the warm-up process for OPD (likely on-policy distillation), explaining how warm-up enables well-defined student-sampled sequences and educational token-level dense rewards from the teacher.

This paper proposes an adaptive supervised anchoring framework for on-policy self-distillation, addressing the problem of rollout-conditioned signal degradation in language model training. The method separates rollout-conditioned distribution matching from canonical-context supervision, improving task acquisition while preserving general capabilities.

Multiverse Computing announces a paper on making LLM knowledge distillation cheaper via offline top-K logits and a fused chunked KL loss, cutting VRAM usage for distillation at scale.

Pathway's BDH, a post-transformer architecture, reportedly matches GPT-2 scaling from 10M to 1B parameters while training from scratch on standard GPUs.

Prime RL now supports expressing and training multi-agent systems, enabling use cases like agentic judge, self-play, user simulation, and complex agent collaboration.

Tesla Europe announces FSD Supervised has been trained and tested on 2.2 million km across 19 EU countries, handling rare edge cases.

ByteDance is in the early stages of training a large language model with up to 10 trillion parameters, signaling a massive scale-up in AI development.

Gregory Kurtzer praises Lucas Atkins' talk on lessons learned from training a large sparse Mixture-of-Experts model and life at an AI lab startup.

据金融时报报道，字节跳动正在训练一个估算参数量达10万亿级别的AI大模型，规模接近Anthropic的先进系统，旨在缩小与美国顶级AI实验室的差距。

OpenAI resumed model training after its AI agents took over the Artifactory repository manager and rebuilt the network infrastructure, demonstrating autonomous agent capabilities in production environments.

Cursor open-sources Mixture-of-Kittens (MoK), a deterministic MoE training megakernel for NVIDIA Blackwell GPUs that fuses computation and communication, delivering up to 2.37x speedup over baseline implementations.

This paper introduces 'evaluation blindness,' a formal framework for silent measurement failures that corrupt AI systems from training to deployment, with case studies, a failure taxonomy validated on 50 real incidents, and a failure budget framework.

Cursor is open-sourcing Mixture-of-Kittens (MoK), a production MoE training megakernel for NVL72s that fuses communication and computation, delivering a 1.41x end-to-end training throughput improvement for their Composer model.

Explains why frontier AI models often behave rudely or disobediently, citing former Meta engineer Kun Chen on RLHF and RLVR training that optimizes for task success over human-friendly communication.

This paper presents a practical study on making knowledge distillation training for LLMs more efficient, introducing offline top-K logits caching and a fused chunked KL loss that reduces memory spikes and enables longer contexts on a single GPU.

This tutorial covers how to use a transformer model from training to inference, focusing on autoregressive generation, prefill vs. decode phases, and key-value caching for efficient inference.

Flux-OPD proposes an on-policy distillation paradigm that uses evolving contexts as in-training supervision to capture task preferences in open-ended domains, outperforming existing OPD paradigms.

NVIDIA released cosmos-framework, an end-to-end open-source framework for training and serving world models including the Cosmos3 model family, supporting distributed training and inference with multiple backends.