@maximelabonne: This is so neat! Dynamic Fine-Tuning (DFT) reweights the SFT loss by the model's own token probability, which creates a…

X AI KOLs Following 05/20/26, 10:31 PM Papers

fine-tuning dynamic-fine-tuning sft kl-divergence deep-learning nlp training-method

Summary

Dynamic Fine-Tuning (DFT) is introduced as a method that reweights the SFT loss using the model's own token probability, creating a feedback loop, and adds forward KL to penalize tokens the base model finds likely but the policy has pushed toward zero probability. The tweet expresses skepticism about SFT papers in practice but praises the attempt.

This is so neat! Dynamic Fine-Tuning (DFT) reweights the SFT loss by the model's own token probability, which creates a feedback loop. So they added forward KL to penalize any token the base finds likely, but the policy has pushed toward zero probability. DFT and forward KL have opposite failure modes, so they kind of fix each other (in theory). SFT papers never work in practice, but I love this attempt!

Original Article

View Cached Full Text

Cached at: 05/21/26, 07:37 PM

This is so neat!

Dynamic Fine-Tuning (DFT) reweights the SFT loss by the model’s own token probability, which creates a feedback loop.

So they added forward KL to penalize any token the base finds likely, but the policy has pushed toward zero probability.

DFT and forward KL have opposite failure modes, so they kind of fix each other (in theory).

SFT papers never work in practice, but I love this attempt!

Similar Articles

On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

Papers with Code Trending

This paper analyzes limitations in standard supervised fine-tuning (SFT) from a reinforcement learning perspective and proposes Dynamic Fine-Tuning (DFT), a simple gradient-rescaling method that improves LLM generalization and matches offline RL performance.

GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification

Hugging Face Daily Papers

GFT (Group Fine-Tuning) is a unified post-training framework for LLMs that addresses limitations of supervised fine-tuning by using Group Advantage Learning and Dynamic Coefficient Rectification to improve training stability and generalization. The paper shows SFT can be interpreted as a special case of policy gradient optimization with sparse implicit rewards, and GFT consistently outperforms SFT-based methods while integrating more smoothly with subsequent RL training.

Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning

arXiv cs.AI

This paper introduces TrimSFT, a token-level reweighting method for supervised fine-tuning that enhances mathematical reasoning by focusing on intermediate logit-gap tokens, demonstrating significant performance gains over standard SFT.

Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models

arXiv cs.AI

Proposes EKSFT, a selective fine-tuning method for large language models that masks tokens with high entropy or high KL divergence from a reference model, preserving pre-trained distribution while injecting task knowledge. Experiments on mathematical reasoning benchmarks show it outperforms standard SFT and improves subsequent RL fine-tuning.

Elite-Weighted Supervised Fine-tuning for Goal-Directed Molecular Optimization

arXiv cs.LG

The paper introduces Elite-Weighted Supervised Fine-tuning (EW-SFT), a method for goal-directed molecular optimization that uses reward to guide elite selection and updates via the model's native loss, applicable across various generative architectures and tasks.