Tag
OpenAI outlines their alignment research approach, highlighting reinforcement learning from human feedback (RLHF) as their primary technique for aligning deployed language models like InstructGPT. They discuss achieving significant preference over 100x larger models while using minimal compute, but acknowledge current limitations and propose a long-term strategy of using AI systems to accelerate alignment research beyond what humans can achieve alone.
OpenAI introduces InstructGPT, a GPT-3 variant fine-tuned using reinforcement learning from human feedback (RLHF) to better follow instructions and reduce harmful outputs. A 1.3B InstructGPT model is preferred by human evaluators over a 175B GPT-3 model, now becoming the default on OpenAI's API.