Tag
The paper identifies manifold drift as a root cause of reward hacking in flow preference optimization and introduces ThermoDPO, a temperature-controlled method that anchors optimization on preferred samples to preserve data manifold integrity and improve performance.