@Orange41324306: TailRL and TailSFT
Summary
Sadhika Malladi proposes TailSFT, a lightweight and principled method to improve coverage and enhance post-RL performance, building on previous research that criticized xent SFT for preparing RL.
View Cached Full Text
Cached at: 09/13/26, 01:11 PM
TailRL and TailSFT
Sadhika Malladi (@SadhikaMalladi): RL is expensive, so every step should count. ~1 yr ago, we showed xent SFT isn’t the best way to prepare for RL (https://t.co/uniFmpbDOn). Now, we propose TailSFT (https://t.co/75yejb8voI), a lightweight + principled way to directly improve coverage and get better post-RL perf.
Similar Articles
On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
This paper analyzes limitations in standard supervised fine-tuning (SFT) from a reinforcement learning perspective and proposes Dynamic Fine-Tuning (DFT), a simple gradient-rescaling method that improves LLM generalization and matches offline RL performance.
SFT or RL for Tool-Calling Agents? A Controlled Study Across Data, Method, and Scale
This controlled study compares supervised fine-tuning and reinforcement learning methods for training tool-calling agents across different datasets and model scales, finding that SFT with LoRA is strongest in-distribution while RL shows slight advantages in cross-dataset transfer.
Tail-Likelihood Reinforcement Learning
The paper proposes Tail-Likelihood Reinforcement Learning (TailRL), an optimization method that focuses on the upper tails of reward distributions to improve policy performance in generative tasks, demonstrated across various applications.
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
This paper investigates the loss of model plasticity after excessive supervised fine-tuning (SFT) in the SFT-then-RL pipeline for LLMs, and proposes Rejuvenation, a method that restores plasticity via base-anchored model fusion and targeted neuron reset, consistently improving RL performance.
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
Harvard researchers challenge the standard LLM training pipeline by showing RL can be effectively applied during pre-training rather than only after SFT, finding that data composition matters more than model scale, and proposing parallel averaging of RL and SFT objectives that outperforms sequential approaches while preserving general capabilities.