@Phoenixyin13: The now-famous PPO, one of the core alignment algorithms underlying ChatGPT, was actually rejected by the top AI conference NIPS (now NeurIPS) in 2017. Even more ironic is the reviewers' reasoning: limited novelty, insufficient improvement over baselines. This phenomenon directly tears off the veil of academia...

The now-famous PPO, one of the core alignment algorithms underlying ChatGPT, was actually rejected by the top AI conference NIPS (now NeurIPS) in 2017. Even more ironic is the reviewers' reasoning: Limited novelty. Insufficient improvement over baselines. This phenomenon directly tears off several layers of disguise from academic peer review. In the academic world, especially in the eyes of top conference reviewers, what counts as innovation? It must involve full pages of incomprehensible mathematical derivations. It must be a completely new and complex framework never proposed before. But the core spirit of PPO is the exact opposite — it is minimalistic. PPO's predecessor is TRPO, a paper with extremely complex mathematical derivations. Although theoretically perfect, TRPO was notoriously hard to implement and computationally expensive. John Schulman's original intention for PPO was to reduce the complex theory into an algorithm that is simple to write, stable to run, and extremely practical for engineering. As a result, this great simplification made for engineering practicality was deemed lacking in innovation by arrogant reviewers. Academia often falls into a self-congratulatory trap: Complicating simple problems is considered skill; simplifying complex problems is considered superficial. Around 2017, reinforcement learning papers were filled with extensive alchemy: Change a random seed, tune a hyperparameter, and experimental results could vary wildly. Many papers would desperately chase benchmarks and tune parameters just to score a high number in a table to please reviewers, but such results were often irreproducible in real industrial settings. On certain specific toy tasks, PPO might indeed not score as high as those specially tuned algorithms. But PPO's true power lies in its generalization and robustness. It was the first RL algorithm that you could simply throw at a complex task without much hyperparameter tuning and get it to work. Reviewers only stared at the benchmarks in front of them, completely unaware that they had summarily rejected an industrial miracle that would later catalyze a multi-billion-dollar large model industry. If John Schulman had doubted his life and abandoned PPO after being rejected by NIPS, perhaps later LLM alignment via RLHF would have taken several more years of detours. Truly great research owes its vitality not to the "Accept" decision of some reviewers.