Text and code embeddings by contrastive pre-training

OpenAI Blog Papers

Summary

OpenAI presents a contrastive pre-training approach for generating high-quality text and code embeddings at scale without supervision, achieving state-of-the-art results on linear-probe classification, semantic search, and code search benchmarks.

No content available
Original Article
View Cached Full Text

Cached at: 04/20/26, 02:56 PM

# Text and code embeddings by contrastive pre-training Source: [https://openai.com/index/text-and-code-embeddings-by-contrastive-pre-training/](https://openai.com/index/text-and-code-embeddings-by-contrastive-pre-training/) ## Abstract Text embeddings are useful features in many applications such as semantic search and computing text similarity\. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture\. In this work, we show that contrastive pre\-training on unsupervised data at scale leads to high quality vector representations of text and code\. The same unsupervised text embeddings that achieve new state\-of\-the\-art results in linear\-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine\-tuned models\. On linear\-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1\.8% over previous best unsupervised and supervised text embedding models respectively\. The same text embeddings when evaluated on large\-scale semantic search attains a relative improvement of 23\.4%, 14\.7%, and 10\.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively\. Similarly to text embeddings, we train code embedding models on \(text, code\) pairs, obtaining a 20\.8% relative improvement over prior best work on code search\.

Similar Articles

Introducing text and code embeddings

OpenAI Blog

OpenAI introduces a new embeddings API endpoint that converts text and code into numerical vector representations for semantic search, clustering, and classification tasks. The models achieve state-of-the-art results on standard benchmarks including a 20% relative improvement in code search performance.

New and improved embedding model

OpenAI Blog

OpenAI released text-embedding-ada-002, a unified embedding model that consolidates five previous models into one with superior performance, 4x longer context (8192 tokens), smaller dimensionality (1536), and 99.8% lower pricing than previous Davinci embeddings.

Improving language understanding with unsupervised learning

OpenAI Blog

OpenAI presents a two-stage approach for improving language understanding: pretraining a transformer model on large unsupervised datasets using language modeling, then fine-tuning on smaller supervised datasets for specific tasks. The method achieves state-of-the-art results across diverse tasks including commonsense reasoning, semantic similarity, and reading comprehension with minimal hyperparameter tuning.

Text-to-Image Generative Modeling Without Language Pretraining

Hugging Face Daily Papers

Halcyon demonstrates that strong text-to-image generation can be trained from scratch on image-text pairs without large-scale language pretraining, outperforming existing models with far less compute. Controlled experiments show text-only pretraining does not improve T2I generation.