Text and code embeddings by contrastive pre-training
Summary
OpenAI presents a contrastive pre-training approach for generating high-quality text and code embeddings at scale without supervision, achieving state-of-the-art results on linear-probe classification, semantic search, and code search benchmarks.
View Cached Full Text
Cached at: 04/20/26, 02:56 PM
Similar Articles
Introducing text and code embeddings
OpenAI introduces a new embeddings API endpoint that converts text and code into numerical vector representations for semantic search, clustering, and classification tasks. The models achieve state-of-the-art results on standard benchmarks including a 20% relative improvement in code search performance.
Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study
This paper empirically studies contrastive pretraining with synthetic semantic supervision for code embeddings in small transformers, showing significant gains over baselines and competitiveness with larger models.
New and improved embedding model
OpenAI released text-embedding-ada-002, a unified embedding model that consolidates five previous models into one with superior performance, 4x longer context (8192 tokens), smaller dimensionality (1536), and 99.8% lower pricing than previous Davinci embeddings.
Improving language understanding with unsupervised learning
OpenAI presents a two-stage approach for improving language understanding: pretraining a transformer model on large unsupervised datasets using language modeling, then fine-tuning on smaller supervised datasets for specific tasks. The method achieves state-of-the-art results across diverse tasks including commonsense reasoning, semantic similarity, and reading comprehension with minimal hyperparameter tuning.
Text-to-Image Generative Modeling Without Language Pretraining
Halcyon demonstrates that strong text-to-image generation can be trained from scratch on image-text pairs without large-scale language pretraining, outperforming existing models with far less compute. Controlled experiments show text-only pretraining does not improve T2I generation.