training

A 117M parameter Silia model was trained on an H100 GPU in 5 hours using 82M tokens. The model is severely under-trained, but comparisons with nanoGPT are provided.

This thread explains GPipe, a paper on pipeline parallelism for scaling large models across multiple GPUs using micro-batching and activation checkpointing.

A user announced they trained a custom 78M parameter AI architecture from scratch using 1B tokens on an 8x3090 rig over 12 hours, claiming the model can now speak.

A multi-agent synthetic data pipeline with SFT and GRPO RL post-training improves HIP compilation and correctness on AMD MI350X GPUs for a 14B open-source model.

Introduces Depth-wise Gradient Augmentation, a general optimization paradigm that transforms block-wise optimizer updates along depth dimension. The method, Gradient Smoothing, improves optimization and generalization across diverse architectures including transformers and diffusion models.

NVIDIA's blog details how FP4, with the NVFP4 format and Blackwell hardware, has evolved from a compression trick to a practical primitive for training and inference across LLMs and diffusion models, achieving near 16-bit accuracy.

This paper proposes Multi-Block Diffusion Language Models (MBD-LMs), extending single-block diffusion to concurrent multi-block decoding with improved training strategies like Multi-block Teacher Forcing and an optimized Block Buffer decoding algorithm. Experiments show increased tokens per forward pass and improved accuracy on benchmarks.

Recommended a GitHub repo for building a GPT-style Transformer from scratch without advanced libraries. With 13M parameters, it can be trained in one day on free Colab to generate grammatically correct text.

MegaTrain enables full-precision training of 100B+ LLMs on a single GPU by treating VRAM as a transient stateless cache, inverting the memory hierarchy.

Progress update on DSpark: training of DFlash backbone and markov head is complete, enabling use on 27B. Next is training the confidence head for adaptive drafting, expected 8-14% speed improvement over DFlash.

The paper identifies position bias in on-policy distillation for language models, where later tokens in student-generated answers receive degraded supervision. The proposed Importance-Weighted On-Policy Distillation (IW-OPD) weights corrections based on accumulated drift, improving learning speed and final performance.

Explains three main approaches to FP8 scaling in LLM pretraining—per-tensor, blockwise, and MXFP8—focusing on how the scale is attached, and derives tile geometries from the constraint that scale must remain constant along the matmul's contracted dimension.

South Korea announces plans to train all 500,000 military personnel to operate drones as a 'universal combat tool,' inspired by drone warfare in Ukraine and the Middle East.

A debugger that detects reward hacking in reinforcement learning reward functions during training, aiding developers in identifying and fixing issues.

Almond Robotics shares one of approximately 1000 training episodes for their robotic system.

Lilian Weng's blog post provides a comprehensive overview of scaling laws in deep learning, covering their derivation, compute-optimal allocation, and the debate between Kaplan et al. and Chinchilla.

You can now train Liquid AI's LFM2-VL model using TRL's GRPO and RLOO methods, with an example script provided.

Nathan Lambert shares a video lecture covering prerequisites for his book, including language model basics, probabilities, and training pipelines, using GLM 5.2.

This paper from OpenAI investigates whether reinforcement learning on beneficial behavior can produce broad and persistent alignment generalization beyond the training distribution. Using a dataset of realistic situations, they show that RL training on beneficial traits improves out-of-distribution alignment and persistence against adversarial attacks.

A live demonstration of an AI agent training a coding agent from a single prompt, with all artifacts recapped.