Tag
TurnTurk-GPT is a text-based turn-taking predictor for Turkish, created by fine-tuning a Turkish GPT-2 model with the TurnGPT objective, which outperforms multilingual end-of-turn detectors and locates Turkish turn-final cues in verb-final morphology.
This paper applies mechanistic interpretability tools—including linear CKA, logit lens, attention decomposition, sparse autoencoders, and activation patching—to compare GPT-2 with DistilGPT-2, revealing a compression-collapse dichotomy where the IOI circuit fully collapses while the Greater-Than circuit survives via layer remapping.
An open-source repository documenting 20 exploratory studies on Next-Concept Modeling (NCM), mechanistic interpretability (such as IOI in GPT-2), and model controls, with full code, results, and transparent reporting of both positive and negative outcomes.
A metro-style map of large model learning designed for Chinese readers: with Transformer→GPT-2→Pre-training→ChatGPT→LLaMA→MoE→Reasoning Models as the main line, branching into five side lines — reasoning, training, alignment, architecture, and applications — emphasizing that hands-on implementation of GPT-2 serves as the anchor for understanding all subsequent models.
该论文研究语言模型如何学习动作名词化结构（ANCs），通过构建 96 种参数化的人工语言并从头训练 GPT-2-small 模型，发现模型学习偏好在基本从句配置上与自然语言类型学分布一致，但在补语系统和 ANC 策略上出现分歧。
回顾2019年《卫报》关于OpenAI因安全顾虑暂缓完整发布GPT-2的报道，作者以调侃语气指出AI发布争议由来已久。
The paper audits attention-head ablation in language models, showing that naive output-level ablation mixes heads via projection and yields nearly uncorrelated results versus corrected pre-projection ablation, while also demonstrating floor/ceiling effects in accuracy and the need for matched controls on held-out data for defensible causal claims.
This preprint uses adversarial training as a controlled instrument on GPT-2 Small to test whether representational simplicity (SAE decomposability, concentrated attribution) implies causal circuit simplicity. It finds that robust models are more SAE-decomposable, while circuit size is regime-dependent: robustness helps at high faithfulness levels (90-95%) but not below 85%.
This paper introduces a method using GPT-2 token surprisal to label telicity in child language, finding that child models rely on syntactic cues like post-verbal determiners, while adult models use semantic features, supporting syntactic bootstrapping theory.
The article humorously or alarmingly discusses the potential dangers and exaggerated capabilities of OpenAI's GPT-2 AI model.
This GitHub repository provides CMake scripts to execute GPT-2 using Q16.16 integer arithmetic, including instructions for running both full and toy models.
A researcher built a mini version of the Kimi-K3 model with 1.02 billion parameters for under $250, achieving a 33.4% HellaSwag score that surpasses GPT-2's 28% score.
Reverse-engineered the RK3588 NPU and built an open compiler and runtime to run GPT-2 and SigLIP from PyTorch, ONNX, and JAX without vendor SDK.
This paper trains five GPT-2-style models from scratch to compare dedicated monolingual models for Tamil, Telugu, Kannada, and Malayalam against a joint multilingual model, finding monolingual models outperform mGPT on sentiment classification and NER with more efficient tokenizers.
Andrej Karpathy released nanochat, a minimalist LLM training framework that can train a GPT-2-level model for just around $48, covering the entire pipeline of pretraining, fine-tuning, and reinforcement learning, with minimal and fully transparent code.
A follow-up project visualizing GPT-2's 32,070 tokens as an interactive hyperbolic tree in a Poincaré ball, allowing users to fly through the embedding space.
An interactive map of GPT-2's token embedding space that lets users tap any token to explore its embeddings.
This paper examines the embedding geometry of GPT-2 Small around the token 'Trump', comparing discretized and continuous nearest neighbor approaches to understand representational structure.
This paper investigates how GPT-2 models pre-trained on impossible languages (with disrupted information locality) can recover natural English, showing a bias toward shorter dependency lengths and dissociation between structural and surface recovery.
The BABEL codec achieves the first complete decode of GPT-2 small's internal state, reconstructing 94.7% of its behavior and enabling reading and writing to the model in English, with open source resources and a demo.