Tag
OpenAI trained the GPT-2 language model but deemed it too dangerous to release to the public due to potential misuse.
This paper argues that current world models lack a persistent state core, proposing a hybrid approach that adds temporal-causal structure via η-pseudo-unitary operator dynamics to convert pretrained GPT-2 into a time-reasoning model.
This paper examines whether language models can independently discover the concept of zero as a form of out-of-distribution generalization, finding that GPT-2 sized models cannot at test time but improve with training on examples of zero, and that language pretraining reduces the number of required examples.
This paper investigates converting pretrained GPT-2 into a time-reasoning model using η-pseudo-unitary operator dynamics, providing mathematical foundations and key findings on PT-breaking transitions and reversible/irreversible sequences.
An article chronicling the timeline of AI model releases since GPT-2, highlighting the accelerating pace of model launches over time.
Proposes MechRL, a reinforcement learning approach to automate circuit discovery in transformer language models. A PPO agent trained on multiple tasks discovers attention head circuits that match known canonical circuits and generalizes to a held-out task.
A developer built AXON, a tool that visualizes GPT-2's internal concept activations as a live 3D force graph using Sparse Autoencoders, allowing users to see interpretable features firing before token generation.
The MAX-LLM book now provides interactive Jupyter notebooks that walk through building a complete GPT-2 implementation from scratch using the MAX framework, enabling users to explore tensor shapes, run components, and generate text.
ArXiv preprint maps stereotype-encoding neurons and attention heads in GPT-2 Small and Llama 3.2, showing biases cluster in small neuron subsets yet ablating them barely reduces biased text generation.
Users are discovering strong meme-generation capabilities in GPT Image 2, particularly for game-specific humor.
Transformer Explainer is an interactive visualization tool that allows non-experts to understand the inner workings of the GPT-2 model through real-time experimentation and visualization in a web browser.
OpenAI's Image GPT (iGPT) applies GPT-2 transformers to pixel sequences for image generation and classification, demonstrating that the same architecture used for language can learn coherent visual features in an unsupervised manner and achieve competitive performance on image classification benchmarks.
OpenAI releases GPT-2 1.5B model with analysis of human perception of credibility, potential for misuse through fine-tuning on extremist ideologies, and challenges in detecting synthetic text. Detection models achieve ~95% accuracy but require complementary approaches for practical deployment.
OpenAI demonstrates fine-tuning GPT-2 (774M parameters) using human preference feedback for text continuation and summarization tasks, requiring 5k labels for stylistic tasks and 60k for summarization, with models achieving 86-88% human preference rates though revealing labeler heuristic exploitation.
OpenAI discusses their 6-month follow-up to GPT-2 release, outlining plans to release the 1558M parameter model in a few months and emphasizing staged release and partnership-based sharing as key to responsible AI publication.
OpenAI introduces GPT-2, a 1.5 billion parameter transformer-based language model trained on 40GB of internet text that achieves state-of-the-art performance on language modeling benchmarks and demonstrates zero-shot capabilities in reading comprehension, translation, question answering, and summarization. Due to safety concerns, only a smaller model and technical paper are released publicly rather than the full trained model.