GPT-2 Small’s embedding geometry around “Trump”: discretized vs. continuous nearest neighbours [P]
Summary
This paper examines the embedding geometry of GPT-2 Small around the token 'Trump', comparing discretized and continuous nearest neighbor approaches to understand representational structure.
Similar Articles
Interactive map of GPT-2's token embedding space - tap any token and explore [P]
An interactive map of GPT-2's token embedding space that lets users tap any token to explore its embeddings.
Geometry of Semantic Space: Comparative Study of Discrete and Continuous Models
This paper compares the geometric structures induced by deep learning vector embeddings (CamemBERT) and lexical co-occurrence graph models on the French 'Great National Debate' corpus, finding similar local topology but distinct global organization, highlighting complementarity between the two approaches.
Follow up: GPT-2's vocabulary as a hyperbolic tree — 32,070 tokens in a Poincaré ball you can fly through [P]
A follow-up project visualizing GPT-2's 32,070 tokens as an interactive hyperbolic tree in a Poincaré ball, allowing users to fly through the embedding space.
MiniGPT: Rebuilding GPT from First Principles
This paper presents MiniGPT, a compact from-scratch implementation of GPT-style autoregressive language modeling in PyTorch, built after studying nanoGPT. It evaluates the model on the Tiny Shakespeare dataset using character-level tokenization, achieving a validation loss of 1.4780 with a 10.77M-parameter configuration.
I built a tool that shows you what GPT-2 is "thinking" in real-time as it generates 3D graph of concept activations per token [R]
A developer built AXON, a tool that visualizes GPT-2's internal concept activations as a live 3D force graph using Sparse Autoencoders, allowing users to see interpretable features firing before token generation.