@maximelabonne: This is so neat! Dynamic Fine-Tuning (DFT) reweights the SFT loss by the model's own token probability, which creates a…
Summary
Dynamic Fine-Tuning (DFT) is introduced as a method that reweights the SFT loss using the model's own token probability, creating a feedback loop, and adds forward KL to penalize tokens the base model finds likely but the policy has pushed toward zero probability. The tweet expresses skepticism about SFT papers in practice but praises the attempt.
View Cached Full Text
Cached at: 05/21/26, 07:37 PM
This is so neat!
Dynamic Fine-Tuning (DFT) reweights the SFT loss by the model’s own token probability, which creates a feedback loop.
So they added forward KL to penalize any token the base finds likely, but the policy has pushed toward zero probability.
DFT and forward KL have opposite failure modes, so they kind of fix each other (in theory).
SFT papers never work in practice, but I love this attempt!
Similar Articles
GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification
GFT (Group Fine-Tuning) is a unified post-training framework for LLMs that addresses limitations of supervised fine-tuning by using Group Advantage Learning and Dynamic Coefficient Rectification to improve training stability and generalization. The paper shows SFT can be interpreted as a special case of policy gradient optimization with sparse implicit rewards, and GFT consistently outperforms SFT-based methods while integrating more smoothly with subsequent RL training.
Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models
Proposes EKSFT, a selective fine-tuning method for large language models that masks tokens with high entropy or high KL divergence from a reference model, preserving pre-trained distribution while injecting task knowledge. Experiments on mathematical reasoning benchmarks show it outperforms standard SFT and improves subsequent RL fine-tuning.
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
This paper investigates the loss of model plasticity after excessive supervised fine-tuning (SFT) in the SFT-then-RL pipeline for LLMs, and proposes Rejuvenation, a method that restores plasticity via base-anchored model fusion and targeted neuron reset, consistently improving RL performance.
@daniel_mac8: babe, wake up. new continual learning breakthrough just dropped. fast-slow training (fst) treats model params as "slow"…
This tweet announces Fast-Slow Training (FST), a new continual learning method that treats model parameters as slow weights and optimized context as fast weights, reportedly outperforming weights-only training on math, code, and general reasoning benchmarks.
Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
This paper introduces a game-theoretic approach to fine-tuning language models that optimizes the trade-off between reward and deviating from a reference policy, providing a principled method for setting the KL regularization coefficient.