Cached at:
04/20/26, 02:55 PM
# Aligning language models to follow instructions
Source: [https://openai.com/index/instruction-following/](https://openai.com/index/instruction-following/)
The[OpenAI API is powered by GPT‑3 language models](https://openai.com/index/gpt-3-apps/)which can be coaxed to perform natural language tasks using carefully engineered text prompts\. But these models can also generate outputs that are untruthful, toxic, or reflect harmful sentiments\. This is in part because GPT‑3 is trained to predict the next word on a large dataset of Internet text, rather than to safely perform the language task that the user wants\. In other words, these models aren’t*aligned*with their users\.
To make our models safer, more helpful, and more aligned, we use an existing technique called[reinforcement learning from human feedback \(RLHF\)](https://openai.com/index/learning-from-human-preferences/)\. On prompts submitted by our customers to the API,[A](https://openai.com/index/instruction-following/#citation-bottom-A)our labelers provide demonstrations of the desired model behavior, and rank several outputs from our models\. We then use this data to fine\-tune GPT‑3\.
The resulting InstructGPT models are much better at following instructions than GPT‑3\. They also make up facts less often, and show small decreases in toxic output generation\. Our labelers prefer outputs from our 1\.3B InstructGPT model over outputs from a 175B GPT‑3 model, despite having more than 100x fewer parameters\. At the same time, we show that we don’t have to compromise on GPT‑3’s capabilities, as measured by our model’s performance on academic NLP evaluations\.
These InstructGPT models, which have been in beta on the API for more than a year, are now the default language models accessible on our API\.[B](https://openai.com/index/instruction-following/#citation-bottom-B)We believe that fine\-tuning language models with humans in the loop is a powerful tool for improving their safety and reliability, and we will continue to push in this direction\.
This is the first time our alignment research, which we’ve been[pursuing](https://openai.com/index/deep-reinforcement-learning-from-human-preferences/)for[several](https://openai.com/index/fine-tuning-gpt-2/)[years](https://openai.com/index/learning-to-summarize-with-human-feedback/),[1](https://openai.com/index/instruction-following/#citation-bottom-1),[2](https://openai.com/index/instruction-following/#citation-bottom-2),[3](https://openai.com/index/instruction-following/#citation-bottom-3)has been applied to our product\. Our work is also related to recent research that fine\-tunes language models to follow instructions using academic NLP datasets, notably FLAN[4](https://openai.com/index/instruction-following/#citation-bottom-4)and T0\.[5](https://openai.com/index/instruction-following/#citation-bottom-5)A key motivation for our work is to increase helpfulness and truthfulness while mitigating the harms and biases of language models\.[6](https://openai.com/index/instruction-following/#citation-bottom-6),[7](https://openai.com/index/instruction-following/#citation-bottom-7),[8](https://openai.com/index/instruction-following/#citation-bottom-8),[9](https://openai.com/index/instruction-following/#citation-bottom-9),[10](https://openai.com/index/instruction-following/#citation-bottom-10)Some of[our previous research](https://openai.com/index/improving-language-model-behavior/)in this direction found that we can reduce harmful outputs by fine\-tuning on a small curated dataset of human demonstrations\.[11](https://openai.com/index/instruction-following/#citation-bottom-11)Other research has focused on filtering the pre\-training dataset,[12](https://openai.com/index/instruction-following/#citation-bottom-12)safety\-specific control tokens,[13](https://openai.com/index/instruction-following/#citation-bottom-13),[14](https://openai.com/index/instruction-following/#citation-bottom-14)or steering model generations\.[15](https://openai.com/index/instruction-following/#citation-bottom-15),[16](https://openai.com/index/instruction-following/#citation-bottom-16)We are exploring these ideas and others in our ongoing alignment research\.
To train InstructGPT models, our core technique is[reinforcement learning from human feedback \(RLHF\)](https://openai.com/index/deep-reinforcement-learning-from-human-preferences/), a method we helped pioneer in our earlier alignment research\. This technique uses human preferences as a reward signal to fine\-tune our models, which is important as the safety and alignment problems we are aiming to solve are complex and subjective, and aren’t fully captured by simple automatic metrics\.
We first collect a dataset of human\-written demonstrations on prompts submitted to our API, and use this to train our supervised learning baselines\. Next, we collect a dataset of human\-labeled comparisons between two model outputs on a larger set of API prompts\. We then train a reward model \(RM\) on this dataset to predict which output our labelers would prefer\. Finally, we use this RM as a reward function and fine\-tune our GPT‑3 policy to maximize this reward using the[PPO algorithm](https://openai.com/index/openai-baselines-ppo/)\.
One way of thinking about this process is that it “unlocks” capabilities that GPT‑3 already had, but were difficult to elicit through prompt engineering alone: this is because our training procedure has a limited ability to teach the model new capabilities relative to what is learned during pretraining, since it uses less than 2% of the compute and data relative to model pretraining\.
A limitation of this approach is that it introduces an “alignment tax”: aligning the models only on customer tasks can make their performance worse on some other academic NLP tasks\. This is undesirable since, if our alignment techniques make models worse on tasks that people care about, they’re less likely to be adopted in practice\. We’ve found a simple algorithmic change that minimizes this alignment tax: during RL fine\-tuning we mix in a small fraction of the original data used to train GPT‑3, and train on this data using the normal log likelihood maximization\.[D](https://openai.com/index/instruction-following/#citation-bottom-D)This roughly maintains performance on safety and human preferences, while mitigating performance decreases on academic tasks, and in several cases even surpassing the GPT‑3 baseline\.