Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features
Summary
This paper proposes SCALE (Selective Control of Adaptation via Local Entropy), an entropy-guided method that freezes the pretrained model and SFT delta while learning bounded token- and module-specific gates to suppress, reverse, or extrapolate learned features during supervised fine-tuning. The approach achieves strong math-reasoning and code-generation results on Qwen2.5-Math and Qwen3-4B-Base models, outperforming existing token-reweighting baselines.
View Cached Full Text
Cached at: 10/05/26, 12:47 PM
Paper page - Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features
Source: https://huggingface.co/papers/2609.33463
Abstract
Supervisedfine-tuning(SFT)learnsmostaggressivelyfromtokensthatthemodeldeemsleastlikely.Thishelpsacquirenewbehaviors,butalsoamplifiesnoisyorconflictingsupervisionandcanoverwriteusefulpretrainedknowledge.Throughaunifiedpolicy-lossview,werevisitexistingtoken-reweightingmethodsandshowthattheyassignnonnegativecoefficientstodemonstratedtokens.Consequently,theycansuppressoramplifysupervisedupdates,butcannotreverseharmfulfeaturesoncelearned.Moreover,largertrainingweightsdonotamounttofeatureextrapolation,sincetheychangetheoptimizationtrajectoryratherthanscaleafixedSFTdirection.WearguethatreversalandextrapolationrequireastablereferenceframedefinedbyafixedSFTdelta.Motivatedbythis,weproposeSCALE(SelectiveControlofAdaptationviaLocalEntropy),anentropy-guidedadaptation-strength-controlmethodthatfreezesthepretrainedmodelandtheSFTdeltaandlearnsboundedtoken-andmodule-specificgatesbyminimizingpredictiveentropyalone.Thesegatessuppress,reverse,orextrapolatefrozenSFTfeaturesaccordingtotheiralignmentwithentropyreduction.AcrossQwen2.5-Math-1.5B,Qwen2.5-Math-7B,andQwen3-4B-Base,SCALEachievesmathematical-reasoningaveragesof37.84,43.60,and36.57,exceedingthestrongestcorrespondingbaselineswhileremainingcompetitiveongeneral-retentionbenchmarks.Italsoattainsthebestaveragecode-generationperformanceacrossHumanEval,HumanEval+,andMBPPforallthreemodels.TheseresultssuggestthateffectiveSFTcorrectioncanbenefitfromcontrollinghowalreadylearnedresidualsareused,ratherthanonlymodifyinghowtheyarelearned.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2609\.33463
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.33463 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.33463 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.33463 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning
This paper introduces TrimSFT, a token-level reweighting method for supervised fine-tuning that enhances mathematical reasoning by focusing on intermediate logit-gap tokens, demonstrating significant performance gains over standard SFT.
@maximelabonne: This is so neat! Dynamic Fine-Tuning (DFT) reweights the SFT loss by the model's own token probability, which creates a…
Dynamic Fine-Tuning (DFT) is introduced as a method that reweights the SFT loss using the model's own token probability, creating a feedback loop, and adds forward KL to penalize tokens the base model finds likely but the policy has pushed toward zero probability. The tweet expresses skepticism about SFT papers in practice but praises the attempt.
Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models
Proposes EKSFT, a selective fine-tuning method for large language models that masks tokens with high entropy or high KL divergence from a reference model, preserving pre-trained distribution while injecting task knowledge. Experiments on mathematical reasoning benchmarks show it outperforms standard SFT and improves subsequent RL fine-tuning.
STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
STARE addresses policy entropy collapse in GRPO-based reinforcement learning for large language models by introducing surprisal-guided token-level advantage reweighting and target-entropy regulation, achieving 4%-8% accuracy gains on AIME benchmarks.
On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
This paper analyzes limitations in standard supervised fine-tuning (SFT) from a reinforcement learning perspective and proposes Dynamic Fine-Tuning (DFT), a simple gradient-rescaling method that improves LLM generalization and matches offline RL performance.