Tag
This paper introduces Retroactive Advantage Correction (RAC), a closed-form bias correction method for delay-aware RLHF that handles asynchronous reward signals by queuing and reinjecting delayed rewards with a V-trace-style clipped residual update.
This paper presents AsyncOPD, a fully asynchronous on-policy distillation pipeline for LLMs, systematically studying the effects of stale-policy data and proposing estimator designs that improve training throughput by 1.6-3.8x while maintaining comparable accuracy.
This paper addresses the missing old logits problem in asynchronous reinforcement learning for LLMs, proposing exact and approximate correction methods to improve training stability and performance.