Summarizing books with human feedback
Summary
OpenAI presents a scalable alignment technique using hierarchical summarization of entire books with human feedback, demonstrating how models can be trained to act in accordance with human intentions on complex, difficult-to-evaluate tasks.
View Cached Full Text
Cached at: 04/20/26, 02:55 PM
Similar Articles
Learning to summarize with human feedback
OpenAI demonstrates a technique for improving language model summarization by training a reward model on human preferences and fine-tuning models with reinforcement learning, achieving significant quality improvements that generalize across datasets. This work advances model alignment through human feedback at scale, with applications beyond summarization.
AI-written critiques help humans notice flaws
OpenAI trained language models to write critiques of text summaries, helping human evaluators spot flaws more effectively — a step toward scalable oversight of AI systems on difficult tasks. The work explores how AI-assisted feedback can improve human evaluation quality as a proof of concept for alignment research.
Our approach to alignment research
OpenAI outlines their alignment research approach, highlighting reinforcement learning from human feedback (RLHF) as their primary technique for aligning deployed language models like InstructGPT. They discuss achieving significant preference over 100x larger models while using minimal compute, but acknowledge current limitations and propose a long-term strategy of using AI systems to accelerate alignment research beyond what humans can achieve alone.
CoAuthorAI: A Human in the Loop System For Scientific Book Writing
CoAuthorAI is a human-in-the-loop system that combines retrieval-augmented generation and hierarchical outlines to enable accurate, coherent scientific book writing, achieving 98% recall and 82% human satisfaction in evaluations.
Towards Human-Level Book-Writing Capability
This paper introduces a dataset and training framework that transforms human-authored novels into multi-resolution planning scaffolds, enabling long-context language models to generate book-scale fiction with more human-like prose and narrative dynamics.