Tag
Benjamin Bai unveils NanoDreamer, a 750K-parameter Dreamer-inspired world-model and actor-critic agent trained entirely on a MacBook M5 Pro, complete with his own autograd engine, tensor library, and transformer, playing all 5 MinAtar games from imagination.
An independent researcher trained Apex-3, a 21B-parameter MoE (3.36B active) from scratch on 70B tokens, releasing a stable checkpoint and sharing training-log evidence of emergent behaviors. The clearest finding is a sharp phase transition in in-context learning around 1.3B-1.5B tokens, while arithmetic gains appear continuous in probability despite looking abrupt in exact match.
An independent developer open-sourced Peacebell, a from-scratch small language model (291M and 148M parameters) specialized in World War II, trained with a custom pipeline on curated synthetic data derived from Wikipedia, alongside a private-question WWII benchmark, demo, and a customized vLLM fork.
An experiment training three from-scratch LLMs with the same GRPO recipe yielded inconsistent results, with GRPO degrading performance in some models, particularly the middle-sized one, and no clear relationship to scale.
A developer trained a 1.1B-parameter LLM from scratch on 20B tokens for about $200, using fineweb-edu for pretraining and LoRA finetuning on OpenHermes. The project includes open-source code, model weights, and a demo website.
Tutorial on implementing a diffusion model based on Google's Gemma architecture from scratch.
This article kicks off a series on building a debugger from scratch in Rust, starting with attaching to a Windows process using OS debugging APIs.
A GitHub repository implementing fundamental machine learning algorithms from scratch using plain NumPy, designed to help learners understand the inner workings of algorithms by focusing on clarity over performance. It covers supervised, unsupervised, deep learning, and reinforcement learning topics.
A 3-hour hands-on walkthrough of building an LLM from scratch, covering five stages: LLM basics, Transformer architecture, training, model modernization, and scaling, with code visible throughout.
A 270M parameter language model trained from scratch on English Wikipedia and instruction-tuned for conversational AI, developed as an independent research project.
Sebastian Raschka announces the release of his book 'Build a Reasoning Model (From Scratch)' after 18 months of work, covering inference scaling, reinforcement learning, and distillation from scratch.
A blog post tutorial on building AI agents from scratch in Python, seeking community feedback.
Recommended a GitHub repo for building a GPT-style Transformer from scratch without advanced libraries. With 13M parameters, it can be trained in one day on free Colab to generate grammatically correct text.
NanoEuler is a GPT-2-scale language model built entirely from scratch in C/CUDA without any ML libraries, including hand-written forward/backward passes, a byte-level BPE tokenizer, and training pipeline. The project is an educational artifact demonstrating the engineering behind transformer training and runs on a single RTX 4070.
A Twitter thread discusses the irony of Andrej Karpathy's teaching philosophy—learning by building from scratch—while he now uses AI to code, and shares a 3-week learning journey using Claude Opus 4.8 to master deep learning the old way.
A developer built a transformer model entirely from scratch in TypeScript, including a custom autograd engine, and released it as an open-source educational tool on GitHub.
A guide on building a large language model from scratch using Apple's MLX framework.
Stanford CS336 aims to teach students how to build language models from scratch, with deep understanding of the full-stack design of data, systems, and models. The course videos are publicly available and suitable for AI beginners.
The author details the process of pretraining and post-training a 500M parameter language model and a 330M parameter image generator entirely from scratch.
A GitHub open-source project that implements the complete GPT training pipeline from scratch, including data preprocessing, pretraining, SFT, and RLHF post-training, all based on native PyTorch. Ideal for developers who want to deeply understand the Transformer architecture.