@rasbt: A little talk on what we can learn from implementing LLM architectures from scratch in Python and PyTorch. And how I ap…

A little talk on what we can learn from implementing LLM architectures from scratch in Python and PyTorch. And how I approach new open-weight models, compare them against reference implementations etc: https://t.co/crKd2l9xGg

TL;DR: The speaker shares hands-on experience implementing LLM architectures in Python/PyTorch: analyzing configs from Hugging Face Model Hub, comparing with reference code, debugging layer by layer to understand the architecture, and emphasizes the importance of the manual process for learning.

Defining “Running LLMs in Python”

When we say running LLMs in Python, we typically mean using the PyTorch library. PyTorch calls C++ (on CPU) or CUDA/ROCm/Metal (on GPU) under the hood, so it acts more like glue between Python and high-speed C++ implementations. Other frameworks like JAX and MLX are also used, but PyTorch is the most popular.

Quick Tour of LLM Ecosystem Tools

• Training phase: Most experiments start with Python + PyTorch; for expensive models (e.g., $50 million training runs), further optimization uses custom CUDA kernels or torch.compile.
• Model storage and exchange: Hugging Face Model Hub is the “GitHub” for open weights; almost all public models are compatible with it.
• Inference tools: Server-side often uses SGLang or VLM; local runs mostly use Llama.cpp or Ollama (also supports MLX). These tools are built on C++ or CUDA, but many ideas still originate in Python.

What You Can Learn from Implementing LLM Architectures from Scratch

Papers these days are often less detailed than in the past; code is the ultimate ground truth. By manually implementing and comparing with references, you can learn many patterns hidden in configs and code.

Personal Workflow

1. Gather information: Learn about new model releases from social media or blogs, find technical reports (e.g., on arXiv) and blog posts to understand design motivations.
2. Check the Model Hub: Go to the Hugging Face page, read the model card and config.json. For example, Gemma 3’s config shows the use of GLU activation, sliding window attention (5 layers full attention + 1 layer sliding window), etc.
3. Compare with old architecture diagrams: Take previously drawn architecture diagrams (e.g., for GPT‑2 XL or Qwen 3) and update them based on the config. For instance, seeing that Gemma 3 uses GeLU instead of SwiGLU means modifying the activation function in the diagram.
4. Code the implementation: Implement the updated architecture in PyTorch, load pretrained weights, and test text generation. If results are abnormal (e.g., garbled output), begin layer‑by‑layer debugging.
5. Compare outputs layer by layer with the reference: Write a script to compare each layer’s output sequentially, starting from the embedding layer, until you locate the discrepancy. For example, the speaker found that the embedding layer was fine, but the first layer’s input normalization (pre‑layer) already showed a difference in tensor mean.

Debugging Case

The speaker shows an LLM running locally with pure Python/PyTorch (interface using Chainlit), from their “Build Reasoning Models from Scratch” repository. When testing Gemma 3 with the same approach, it generated some incomprehensible foreign tokens, indicating a bug that required layer‑by‑layer investigation.

AI Trends and Getting Started (Brief Mention)

The speaker originally planned to discuss current LLM architecture trends and a recommended project roadmap (from beginner to expert), but the transcript does not expand on this. Related resources can be found in their repository and open‑source projects.

Source: @rasbt: A little talk on what we can learn from implementing LLM architectures from scratch in Python and PyTorch. And how I ap… (https://www.youtube.com/watch?v=TXzQ7PGpO6w)