gpt-2

This paper trains five GPT-2-style models from scratch to compare dedicated monolingual models for Tamil, Telugu, Kannada, and Malayalam against a joint multilingual model, finding monolingual models outperform mGPT on sentiment classification and NER with more efficient tokenizers.

Andrej Karpathy released nanochat, a minimalist LLM training framework that can train a GPT-2-level model for just around $48, covering the entire pipeline of pretraining, fine-tuning, and reinforcement learning, with minimal and fully transparent code.

A follow-up project visualizing GPT-2's 32,070 tokens as an interactive hyperbolic tree in a Poincaré ball, allowing users to fly through the embedding space.

An interactive map of GPT-2's token embedding space that lets users tap any token to explore its embeddings.

This paper examines the embedding geometry of GPT-2 Small around the token 'Trump', comparing discretized and continuous nearest neighbor approaches to understand representational structure.

This paper investigates how GPT-2 models pre-trained on impossible languages (with disrupted information locality) can recover natural English, showing a bias toward shorter dependency lengths and dissociation between structural and surface recovery.

The BABEL codec achieves the first complete decode of GPT-2 small's internal state, reconstructing 94.7% of its behavior and enabling reading and writing to the model in English, with open source resources and a demo.

This paper introduces Manifestation Units, a typed tuple protocol for organizing per-component statistics from mechanistic interpretability analyses into structured, queryable fields. The protocol is demonstrated across vision (β-VAE, CNN) and language (GPT-2) models, showing improved retrieval and causal sufficiency.

This paper re-derives activation patching from causal mediation analysis, revealing that the natural indirect effect (NIE) captures not only a component's causal effect but also interaction effects with other components. It demonstrates these hidden interactions in the GPT-2 IOI circuit and argues that they are a diagnostic tool rather than a nuisance.

NanoEuler is a GPT-2-scale language model built entirely from scratch in C/CUDA without any ML libraries, including hand-written forward/backward passes, a byte-level BPE tokenizer, and training pipeline. The project is an educational artifact demonstrating the engineering behind transformer training and runs on a single RTX 4070.

OpenAI trained the GPT-2 language model but deemed it too dangerous to release to the public due to potential misuse.

This paper argues that current world models lack a persistent state core, proposing a hybrid approach that adds temporal-causal structure via η-pseudo-unitary operator dynamics to convert pretrained GPT-2 into a time-reasoning model.

This paper examines whether language models can independently discover the concept of zero as a form of out-of-distribution generalization, finding that GPT-2 sized models cannot at test time but improve with training on examples of zero, and that language pretraining reduces the number of required examples.

This paper investigates converting pretrained GPT-2 into a time-reasoning model using η-pseudo-unitary operator dynamics, providing mathematical foundations and key findings on PT-breaking transitions and reversible/irreversible sequences.

An article chronicling the timeline of AI model releases since GPT-2, highlighting the accelerating pace of model launches over time.

Proposes MechRL, a reinforcement learning approach to automate circuit discovery in transformer language models. A PPO agent trained on multiple tasks discovers attention head circuits that match known canonical circuits and generalizes to a held-out task.

A developer built AXON, a tool that visualizes GPT-2's internal concept activations as a live 3D force graph using Sparse Autoencoders, allowing users to see interpretable features firing before token generation.

The MAX-LLM book now provides interactive Jupyter notebooks that walk through building a complete GPT-2 implementation from scratch using the MAX framework, enabling users to explore tensor shapes, run components, and generate text.

ArXiv preprint maps stereotype-encoding neurons and attention heads in GPT-2 Small and Llama 3.2, showing biases cluster in small neuron subsets yet ablating them barely reduces biased text generation.

Users are discovering strong meme-generation capabilities in GPT Image 2, particularly for game-specific humor.