Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning

# Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning
Source: [https://arxiv.org/html/2609.09707](https://arxiv.org/html/2609.09707)
Chunhui ZhangAffiliation:Dartmouth CollegeWenxuan XuAffiliation:Dartmouth CollegeXingjian DiaoAffiliation:Dartmouth CollegeXiaoyuan WangAffiliation:Carnegie Mellon Universityyaning\.jia\.gr@dartmouth\.edusoroush\.vosoughi@dartmouth\.eduSoroush Vosoughi††thanks:Correspondence tosoroush\.vosoughi@dartmouth\.eduAffiliation:Dartmouth College

###### Abstract

Supervised fine\-tuning \(SFT\) applies a uniform cross\-entropy loss to all target tokens, even though different tokens provide unequal learning signals for mathematical reasoning\. This uniform treatment can over\-sharpen already mastered tokens while amplifying learning pressure on uncertain, low\-confidence tokens, leading to suboptimal training dynamics\. We proposeTrimmed Logit\-Gap SFT \(TrimSFT\), a simple token\-level reweighting method that scales the SFT loss according to the logit gap between the gold token and its strongest competitor\. TrimSFT trims supervision away from both extremes: tokens already mastered \(large logit gap\) and tokens weakly supported by the current model \(small or negative logit gap\), concentrating learning within an intermediate logit\-gap region between them\. We instantiate this principle with a Gaussian weight centered at marginmmwith bandwidthτ\\tau, requiring no reference model or additional forward pass\. We evaluate TrimSFT on six base models from the Llama, Qwen, and DeepMath families across five mathematical reasoning benchmarks\. TrimSFT consistently improves over standard SFT, achieving the best average performance on five out of six models, with gains of up to\+26\.9\+26\.9points over SFT on MATH500\. Further analyses show that the bandwidthτ\\taumatters more than the exact margin location, and that half\-trim variants that remove supervision pressure from only one side yield inferior trade\-offs\. A token\-level logit\-gap distribution analysis suggests that TrimSFT reshapes model confidence in a more balanced way than uniform SFT or monotonic reweighting methods\. These results suggest that reasoning SFT can benefit from trimming both extremes rather than treating all tokens uniformly\.

## 1Introduction

Large language models \(LLMs\) have demonstrated strong capabilities on complex reasoning tasks\([Wei et al\., 2022b](https://arxiv.org/html/2609.09707#bib.bib1);[Wang et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib2);[Fan et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib3)\), with mathematical reasoning serving as a key testbed for studying multi\-step inference\([Jia et al\., 2025](https://arxiv.org/html/2609.09707#bib.bib4);[Liu et al\., 2025](https://arxiv.org/html/2609.09707#bib.bib5)\)\. Supervised fine\-tuning \(SFT\) is widely used to adapt pretrained models to mathematical reasoning data\([Yu et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib6);[Yue et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib7)\), and often serves as an initialization stage for downstream reinforcement learning or preference optimization\([Ouyang et al\., 2022](https://arxiv.org/html/2609.09707#bib.bib10);[Bai et al\., 2022](https://arxiv.org/html/2609.09707#bib.bib11)\)\.

Despite its effectiveness, standard SFT applies token\-level cross\-entropy with uniform weighting across all target tokens\([Lin et al\., 2026](https://arxiv.org/html/2609.09707#bib.bib9)\), ignoring that different tokens can provide very different learning signals\([Wu et al\., 2026](https://arxiv.org/html/2609.09707#bib.bib8);[Gong et al\., 2026](https://arxiv.org/html/2609.09707#bib.bib12)\)\. In reasoning trajectories, some tokens may already be well mastered and continue to receive unnecessary sharpening pressure, which can contribute to over\-confidence\([Pereyra et al\., 2017](https://arxiv.org/html/2609.09707#bib.bib14);[Chen et al\., 2025](https://arxiv.org/html/2609.09707#bib.bib15);[Wei et al\., 2022a](https://arxiv.org/html/2609.09707#bib.bib16)\)\. Other tokens may be highly uncertain, noisy, or beyond the model’s current capability, yet still induce large losses and dominate the optimization signal\. These two extremes suggest that treating all tokens uniformly can pull learning away from a potentially useful intermediate region\([Wu et al\., 2026](https://arxiv.org/html/2609.09707#bib.bib8);[Lin et al\., 2017](https://arxiv.org/html/2609.09707#bib.bib17);[Han et al\., 2018](https://arxiv.org/html/2609.09707#bib.bib18)\)\. This motivates a selective training objective that reduces supervision at both ends of the logit\-gap spectrum while concentrating learning on an intermediate region\.

We therefore proposeTrimmed Logit\-Gap SFT \(TrimSFT\), a token\-level reweighting method that trims supervision away from both extremes: tokens already mastered \(large logit gap\) and tokens weakly supported by the current model \(small or negative logit gap\)\. For each target token, we compute its*logit gap*, defined as the margin between the gold token’s logit and that of its strongest competitor, and use a Gaussian weight centered at marginmmwith bandwidthτ\\tauto scale the cross\-entropy loss \(Figure[1](https://arxiv.org/html/2609.09707#S1.F1), left\)\. Tokens whose logit gaps lie nearmmreceive stronger supervision, while tokens with much smaller or larger gaps are softly down\-weighted, concentrating learning within a bounded intermediate region of the logit\-gap spectrum\. The weight is computed from the model’s own logits in the same forward pass, requiring no reference model or additional forward pass\. To examine the contribution of each side of this two\-sided trimming profile, we further introduce two half\-trim variants \(Figure[1](https://arxiv.org/html/2609.09707#S1.F1), right\):Trim\-Easy SFT \(TrimSFT\-E\), which trims the hard side only and preserves full weight on high\-gap \(easy\) tokens, andTrim\-Hard SFT \(TrimSFT\-H\), which trims the easy side only and preserves full weight on low\-gap \(hard\) tokens\.

Figure 1:Token weighting functions for SFT, TrimSFT, and its half\-trim variants TrimSFT\-E and TrimSFT\-H\. The marginmmdetermines the center of the weighting function, andτ\\taucontrols the width of the weighted region\.We evaluate TrimSFT on six base models from the Llama, Qwen, and DeepSeekMath families across five mathematical reasoning benchmarks\. Our results show that TrimSFT consistently improves over standard SFT, and we further analyze the mechanism behind these gains\. Our contributions are summarized as follows:

- •We introduceTrimSFT, a token\-level reweighting method that scales the SFT loss by the logit gap between the gold token and its strongest competitor\. It requires no reference model or additional forward pass\.
- •TrimSFT achieves the best average performance on five out of six models, with gains of up to\+26\.9\+26\.9points over SFT on MATH500\. It also improves capability coverage and self\-consistency under repeated sampling, as measured by pass@8 and best\-of\-8, respectively\.
- •Ablations show that performance is more sensitive to the bandwidthτ\\tauthan to the marginmm, suggesting that the width of the selected logit\-gap region is more important than its center\.
- •Half\-trim variants show that trimming only one side yields inferior trade\-offs: TrimSFT\-H collapses below SFT, and TrimSFT\-E degrades on the hardest problems\. Token\-level logit\-gap distribution analysis further shows that TrimSFT reshapes model confidence in a more balanced way than uniform SFT or monotonic reweighting methods\.

## 2Related Work

#### Supervised Fine\-Tuning for Mathematical Reasoning\.

Mathematical reasoning has emerged as a central testbed for evaluating the multi\-step inference capabilities of large language models\([Cobbe et al\., 2021](https://arxiv.org/html/2609.09707#bib.bib19);[Hendrycks et al\., 2021](https://arxiv.org/html/2609.09707#bib.bib20);[He et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib21);[Saxton et al\., 2019](https://arxiv.org/html/2609.09707#bib.bib35);[Lewkowycz et al\., 2022](https://arxiv.org/html/2609.09707#bib.bib36)\)\. A common recipe for adapting pretrained models to mathematical tasks is supervised fine\-tuning \(SFT\) on curated reasoning trajectories\([Yu et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib6);[Yue et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib7);[Toshniwal et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib22);[Li et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib23)\)\. Much of the progress in this line has come from data\-centric improvements, including synthesizing high\-quality chain\-of\-thought solutions\([Yu et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib6);[Luo et al\., 2025](https://arxiv.org/html/2609.09707#bib.bib24)\), distilling from stronger teachers\([Shao et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib25);[Yang et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib26)\), incorporating recent long chain\-of\-thought trajectories from reasoning\-specialized models\([Guo et al\., 2025](https://arxiv.org/html/2609.09707#bib.bib27);[Hugging Face, 2025](https://arxiv.org/html/2609.09707#bib.bib28);[Ye et al\., 2025](https://arxiv.org/html/2609.09707#bib.bib29)\), and constructing curated datasets through difficulty\- or correctness\-based filtering\([Toshniwal et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib22);[Li et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib23)\)\. Despite these advances, the underlying objective typically remains the standard token\-level cross\-entropy loss applied uniformly to every token\. In contrast, our work targets the per\-token objective rather than the training data, making it complementary to existing data\-centric approaches\.

#### Token\-Level Reweighting and Selection in SFT\.

While vanilla SFT applies a uniform cross\-entropy loss across all tokens, recent work has explored token\-level reweighting or selection to account for differences in training value across tokens\([Lin et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib30);[Ruan et al\., 2025](https://arxiv.org/html/2609.09707#bib.bib31);[Wu et al\., 2026](https://arxiv.org/html/2609.09707#bib.bib8)\)\. These methods differ mainly in the signals used to score tokens and in the form of loss modulation\. Some approaches rely on auxiliary signals or pre\-computed token masks, such as reference\-model\-based scoring in Rho\-1\([Lin et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib30)\)and counterfactual selection in CFT\([Ruan et al\., 2025](https://arxiv.org/html/2609.09707#bib.bib31)\)\. DFT\([Wu et al\., 2026](https://arxiv.org/html/2609.09707#bib.bib8)\)and Focal Loss\([Lin et al\., 2017](https://arxiv.org/html/2609.09707#bib.bib17)\)both rescale token\-level cross\-entropy using the predicted probability of the gold token, though in opposite monotonic directions: DFT up\-weights tokens already assigned high probability, while Focal Loss down\-weights them\. Our method departs from these probability\-based schemes in two ways: it uses the*logit gap*as a more scale\-sensitive signal, and applies a Gaussian band\-pass weighting that targets a bounded intermediate region rather than following a monotonic trend\. We elaborate on this comparison in Section[3\.3](https://arxiv.org/html/2609.09707#S3.SS3)\.

#### Logit Gaps, Margins, and Confidence Shaping\.

Margin\-related quantities have been widely used to shape confidence and compare competing predictions, but they are rarely used directly as token\-level supervision weights\. In classification calibration, label smoothing\([Müller et al\., 2019](https://arxiv.org/html/2609.09707#bib.bib32);[Pereyra et al\., 2017](https://arxiv.org/html/2609.09707#bib.bib14)\)and margin\-based label smoothing\([Liu et al\., 2022](https://arxiv.org/html/2609.09707#bib.bib33)\)shape confidence and margin behavior to mitigate overconfidence, while logit normalization\([Wei et al\., 2022a](https://arxiv.org/html/2609.09707#bib.bib16)\)and calibration\-oriented training objectives\([Guo et al\., 2017](https://arxiv.org/html/2609.09707#bib.bib37)\)regulate confidence and logit magnitude during training\. Related concerns have also been studied in large language models, where recent work examines confidence calibration and overconfidence in generated outputs\([Zhang et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib38);[Leng et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib39)\)\. Recent evidence further shows that linguistic confidence can diverge substantially from internal, logit\-based confidence, highlighting the importance of distinguishing verbalized confidence from model\-internal confidence signals\([Zhang et al\., 2026](https://arxiv.org/html/2609.09707#bib.bib13)\)\. In preference optimization, methods such as SimPO\([Meng et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib34)\)use sequence\-level logit margins between preferred and rejected outputs\. These works use margin\-related quantities either to regulate confidence or to define sequence\-level preference signals\. In contrast, TrimSFT uses the per\-token logit gap as a supervision weight during SFT, trimming both extremes of the token\-confidence spectrum rather than constraining or maximizing margins globally\.

## 3Method

We begin from the standard supervised fine\-tuning \(SFT\) objective and then introduce TrimSFT, a token\-level reweighting scheme driven by the model’s own logit gaps\. The key idea is to allocate stronger supervision to tokens whose logits place them near a bounded decision region, while reducing learning pressure on tokens that are already well mastered or currently beyond the model’s effective reach\. TrimSFT deliberately reduces supervision pressure at both ends of the logit\-gap spectrum: tokens with very large gaps \(already mastered\) receive lower weight, as do tokens with very small or negative gaps \(weakly supported by the current model\)\.

Given an input promptxxand a target token sequencey1,…,yTy\_\{1\},\\dots,y\_\{T\}, standard SFT trains a modelπθ\\pi\_\{\\theta\}by minimizing the token\-level cross\-entropy loss

ℒSFT=−∑t=1Tlogπθ\(yt∣x,y<t\),\\mathcal\{L\}\_\{\\text\{SFT\}\}=\-\\sum\_\{t=1\}^\{T\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\),\(1\)which assigns uniform weight to every target token\. Letzt∈ℝVz\_\{t\}\\in\\mathbb\{R\}^\{V\}denote the pre\-softmax logits at positiontt, and letzt,ytz\_\{t,y\_\{t\}\}denote the logit assigned to the gold tokenyty\_\{t\}\. We define the*logit gap*at positionttas

Δt=zt,yt−maxv≠yt⁡zt,v,\\Delta\_\{t\}=z\_\{t,y\_\{t\}\}\-\\max\_\{v\\neq y\_\{t\}\}z\_\{t,v\},\(2\)namely, the margin between the gold token’s logit and that of its strongest competing token\. A positiveΔt\\Delta\_\{t\}indicates that the gold token is currently the top\-1 prediction, while a larger magnitude reflects greater separation from the closest alternative\. This quantity provides a local, token\-level measure of how decisively the model distinguishes the correct token from competing candidates\.

### 3\.1Trimmed Logit\-Gap SFT

Our goal is to trim supervision away from both extremes of the logit\-gap spectrum and concentrate learning on tokens whose gaps fall within an intermediate region\. Intuitively, tokens with very large positive gaps are already well separated and may benefit little from continued sharpening, whereas tokens with very small or negative gaps may be highly uncertain, unstable, or beyond the model’s current capability\. We therefore seek a weighting function that peaks around a moderate logit\-gap region and decays on both sides\.

We instantiate this with a Gaussian weight on the logit gap:

wt=exp⁡\(−\(Δt−m\)22​τ2\),w\_\{t\}=\\exp\\\!\\left\(\-\\frac\{\(\\Delta\_\{t\}\-m\)^\{2\}\}\{2\\tau^\{2\}\}\\right\),\(3\)wheremmis the center of the weighting function andτ\\taucontrols the width of the high\-weight region\. Tokens whose logit gaps lie nearmmreceive the largest weights, while tokens whose gaps are substantially smaller or larger are smoothly down\-weighted\. In this sense,mmdetermines*where*learning should be concentrated, andτ\\taudetermines*how broadly*that concentration should spread\.

The resulting TrimSFT objective is

ℒTrimSFT=−∑t=1Tsg\(wt\)logπθ\(yt∣x,y<t\),\\mathcal\{L\}\_\{\\text\{TrimSFT\}\}=\-\\sum\_\{t=1\}^\{T\}\\operatorname\{sg\}\(w\_\{t\}\)\\,\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\),\(4\)wheresg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\)denotes stop\-gradient\. The weightswtw\_\{t\}are computed from the same forward pass as the logits and then detached, so they act purely as a rescaling per\-token of the cross\-entropy loss without contributing a gradient throughθ\\theta\. This does not require any auxiliary model or extra forward pass, and adds only minimal computational overhead\.

Compared with probability\-based reweighting, the logit gap is particularly suitable in our setting for two reasons\. First, it directly measures the separation between the gold token and its strongest competitor, making it naturally aligned with how close a token is to the model’s decision boundary\. Second, unlike the gold\-token probability, which tends to saturate in high\-confidence regimes, the logit gap remains discriminative even when probabilities would otherwise appear nearly indistinguishable\. The Gaussian weighting then realizes a smooth two\-sided trimming profile over token states: supervision is concentrated on tokens whose logit gaps lie near the chosen marginmm, and decays on both sides\. Unlike monotonic reweighting schemes that progressively emphasize either harder or easier tokens across the full confidence range, TrimSFT explicitly trims both extremes and focuses learning on a bounded intermediate region\.

### 3\.2Half\-Trim Variants

The weighting function in TrimSFT is symmetric around the marginmm, assigning lower weights to tokens on both sides as their logit gaps move away frommm\. To isolate the contribution of each side, we define two half\-trim variants that retain Gaussian decay on only one side while keeping full weight on the other \(Figure[1](https://arxiv.org/html/2609.09707#S1.F1), right\)\.

#### TrimSFT\-E\.

TrimSFT\-E trims the hard side only: it keeps full weight on tokens whose logit gaps exceed the marginmm, and applies Gaussian decay only on the lower\-gap side:

wtE=\{exp⁡\(−\(Δt−m\)22​τ2\),Δt<m,1,Δt≥m\.w\_\{t\}^\{\\text\{E\}\}=\\begin\{cases\}\\exp\\\!\\left\(\-\\dfrac\{\(\\Delta\_\{t\}\-m\)^\{2\}\}\{2\\tau^\{2\}\}\\right\),&\\Delta\_\{t\}<m,\\\\\[6\.0pt\] 1,&\\Delta\_\{t\}\\geq m\.\\end\{cases\}\(5\)This preserves full supervision for tokens already beyond the chosen margin, while down\-weighting tokens with smaller gaps\.

#### TrimSFT\-H\.

TrimSFT\-H trims the easy side only: it keeps full weight on tokens whose logit gaps fall below the marginmm, and applies Gaussian decay only on the higher\-gap side:

wtH=\{1,Δt≤m,exp⁡\(−\(Δt−m\)22​τ2\),Δt\>m\.w\_\{t\}^\{\\text\{H\}\}=\\begin\{cases\}1,&\\Delta\_\{t\}\\leq m,\\\\\[6\.0pt\] \\exp\\\!\\left\(\-\\dfrac\{\(\\Delta\_\{t\}\-m\)^\{2\}\}\{2\\tau^\{2\}\}\\right\),&\\Delta\_\{t\}\>m\.\\end\{cases\}\(6\)This preserves full supervision for tokens with smaller gaps, while down\-weighting tokens that are already well separated\.

Together, these variants retain the same marginmmand bandwidthτ\\tauas TrimSFT, but isolate the contribution of each side of the weighting profile\. Comparing them with full TrimSFT allows us to determine whether its gains arise primarily from trimming high\-gap tokens, low\-gap tokens, or both\.

### 3\.3A Unified View of Token Reweighting

Standard SFT, TrimSFT, and several representative reweighting methods, including DFT\([Wu et al\., 2026](https://arxiv.org/html/2609.09707#bib.bib8)\)and a focal\-loss\-based SFT baseline \(FSFT\)\([Lin et al\., 2017](https://arxiv.org/html/2609.09707#bib.bib17)\), can all be written in a unified token\-reweighted form:

ℒ=∑t=1Tsg⁡\(wt\)​ℓt,\\mathcal\{L\}=\\sum\_\{t=1\}^\{T\}\\operatorname\{sg\}\(w\_\{t\}\)\\,\\ell\_\{t\},\(7\)whereℓt=−log⁡πθ​\(yt∣x,y<t\)\\ell\_\{t\}=\-\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)is the standard token\-level cross\-entropy loss,sg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\)denotes stop\-gradient, and the methods differ only in the choice ofwtw\_\{t\}\. Representative objectives then correspond to different weighting functions:

wt=\{1,SFT,\(1−pt,yt\)γ,FSFT,pt,yt,DFT,exp⁡\(−\(Δt−m\)22​τ2\),TrimSFT\.w\_\{t\}=\\begin\{cases\}1,&\\text\{SFT\},\\\\\[4\.0pt\] \(1\-p\_\{t,y\_\{t\}\}\)^\{\\gamma\},&\\text\{FSFT\},\\\\\[4\.0pt\] p\_\{t,y\_\{t\}\},&\\text\{DFT\},\\\\\[4\.0pt\] \\exp\\\!\\left\(\-\\dfrac\{\(\\Delta\_\{t\}\-m\)^\{2\}\}\{2\\tau^\{2\}\}\\right\),&\\text\{TrimSFT\}\.\\end\{cases\}\(8\)
In this framework, standard SFT assigns equal training weight to every token\. FSFT and DFT instead reshape the SFT objective using the gold\-token probability: FSFT places larger weights on lower\-confidence tokens, thereby emphasizing harder or less well\-mastered positions, whereas DFT increases the relative contribution of higher\-probability tokens to compensate for the implicit inverse\-probability bias of standard cross\-entropy\. In contrast, TrimSFT uses the logit gapΔt\\Delta\_\{t\}and applies a non\-monotonic weighting profile that trims both extremes while concentrating learning on tokens whose gaps lie near a prescribed marginmm\.

This unified view makes two differences especially clear\. First, TrimSFT operates on the logit gap rather than the probability, so it directly tracks the separation between the gold token and its strongest competitor and avoids the saturation that compresses high\-confidence tokens to nearly identical weights under probability\-based schemes\. Second, TrimSFT targets a bounded intermediate region between well\-mastered and weakly supported tokens, whereas existing probability\-based reweighting methods follow a single monotonic trend over the full confidence range\.

## 4Experiments

### 4\.1Setup

#### Dataset\.

NuminaMath\-CoT\([Li et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib23)\)is a large\-scale math reasoning dataset containing about 860K problem\-solution pairs with chain\-of\-thought annotations\. For all experiments, we randomly sample 20,000 problems for supervised fine\-tuning\.

#### Models\.

We evaluate TrimSFT and the baseline methods on six base models from three families, spanning parameter scales from 1\.5B to 8B: Llama3\.2\-3B\([Meta AI, 2024](https://arxiv.org/html/2609.09707#bib.bib40)\), Llama3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib41)\), DeepSeekMath\-7B\([Shao et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib25)\), Qwen2\.5\-Math\-1\.5B, Qwen2\.5\-Math\-7B\([Yang et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib26)\), and Qwen3\-4B\-Base\([Yang et al\., 2025](https://arxiv.org/html/2609.09707#bib.bib42)\)\. We use only base models, rather than instruction\-tuned variants, to reduce the influence of prior instruction tuning and enable a cleaner comparison of different supervised fine\-tuning objectives\.

#### Baselines\.

We compareTrimSFTwith four representative baselines: \(i\)Base, the original pretrained model without supervised fine\-tuning; \(ii\)SFT, standard supervised fine\-tuning with uniform token\-level weighting; \(iii\)FSFT, a focal\-loss\-based variant that upweights lower\-confidence tokens during training\([Lin et al\., 2017](https://arxiv.org/html/2609.09707#bib.bib17)\); and \(iv\)DFT\([Wu et al\., 2026](https://arxiv.org/html/2609.09707#bib.bib8)\), a probability\-based reweighting approach that assigns larger weights to higher\-probability tokens\.

#### Evaluation benchmarks\.

We evaluate all methods on five mathematical reasoning benchmarks spanning a range of difficulty levels: MATH500\([Hendrycks et al\., 2021](https://arxiv.org/html/2609.09707#bib.bib20)\), OlympiadBench\([He et al\., 2024](https://arxiv.org/html/2609.09707#bib.bib21)\), Minerva\([Lewkowycz et al\., 2022](https://arxiv.org/html/2609.09707#bib.bib36)\), AMC\([AI\-MO, 2024b](https://arxiv.org/html/2609.09707#bib.bib44)\), and AIME24\([AI\-MO, 2024a](https://arxiv.org/html/2609.09707#bib.bib43)\)\. For each benchmark, we sampleN=8N=8generations per problem and reportaverage@8as the primary metric, which reflects the model’s average generation quality\. In Section[4\.3](https://arxiv.org/html/2609.09707#S4.SS3), we additionally reportpass@8andbest\-of\-8, where majority voting is used as the selector, to characterize the model’s exploration ability and self\-consistency, respectively\. Detailed evaluation settings are provided in Appendix[A](https://arxiv.org/html/2609.09707#A1)\.

#### Implementation details\.

For TrimSFT, we use a single hyperparameter setting,\(m,τ\)=\(1\.5,0\.8\)\(m,\\tau\)=\(1\.5,0\.8\), for all models and benchmarks in the main comparison \(Table[1](https://arxiv.org/html/2609.09707#S4.T1)\), without any model\- or benchmark\-specific tuning\. All results are reported using the checkpoint obtained after one training epoch\. We study the effects of varyingmmandτ\\tauin Sections[4\.4](https://arxiv.org/html/2609.09707#S4.SS4)and[4\.5](https://arxiv.org/html/2609.09707#S4.SS5), and provide additional training and evaluation details in Appendix[A](https://arxiv.org/html/2609.09707#A1)\. Code is available at[https://github\.com/karpning/TrimSFT](https://github.com/karpning/TrimSFT)\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2609.09707#S4.T1)reports the main results under theaverage@8accuracy across six base models and five mathematical reasoning benchmarks, with TrimSFT evaluated using\(m,τ\)=\(1\.5,0\.8\)\(m,\\tau\)=\(1\.5,0\.8\)\.

Table 1:Average@8 accuracy \(%\) on math reasoning benchmarks\.Boldmarks the best result andunderlinedmarks the second\-best within each model block\. Avg\. denotes the mean across benchmarks\.Overall, TrimSFT consistently improves over standard SFT across all six models and achieves the best average performance on five out of six models\. The gains are especially large on math\-oriented models: for example, TrimSFT improves the average score from15\.4715\.47to31\.7731\.77on Qwen2\.5\-Math\-1\.5B and from20\.9520\.95to35\.9835\.98on Qwen2\.5\-Math\-7B\. On MATH500, TrimSFT is the top\-performing method across all six models, with the largest gain over SFT reaching\+26\.93\+26\.93points on Qwen2\.5\-Math\-1\.5B \(40\.02→66\.9540\.02\\rightarrow 66\.95\)\. Compared with DFT, TrimSFT achieves stronger overall average performance on five of the six models, suggesting that logit\-gap\-based reweighting provides a useful alternative to probability\-based token weighting\. In contrast, FSFT performs poorly in most settings, indicating that simply emphasizing harder tokens is insufficient for reasoning SFT\. These results provide initial evidence for our hypothesis that effective supervised fine\-tuning for mathematical reasoning benefits from trimming both extremes of the logit\-gap spectrum rather than treating all tokens uniformly\.

### 4\.3Capability Ceiling and Self\-Consistency

Beyondaverage@8, we further evaluatepass@8andbest\-of\-8with majority voting to characterize two complementary aspects of model behavior\. Pass@8 measures whether the model can produce at least one correct solution among multiple samples, reflecting its capability ceiling or exploration coverage\. Best\-of\-8 with majority voting measures whether the model’s sampled solutions consistently support the correct answer, reflecting the self\-consistency of its generation distribution\.

We report results on Qwen2\.5\-Math\-1\.5B and Llama3\.1\-8B in Figure[2](https://arxiv.org/html/2609.09707#S4.F2)\. For both models, TrimSFT uses the same fixed hyperparameter setting\(m,τ\)=\(1\.5,0\.8\)\(m,\\tau\)=\(1\.5,0\.8\)as in the main comparison in Table[1](https://arxiv.org/html/2609.09707#S4.T1)\.

Figure 2:Pass@8 and best\-of\-8 with majority voting on Qwen2\.5\-Math\-1\.5B and Llama3\.1\-8B across five mathematical reasoning benchmarks\.As shown in Figure[2](https://arxiv.org/html/2609.09707#S4.F2), TrimSFT consistently improves pass@8 over standard SFT on both models across most benchmarks, indicating that two\-sided logit\-gap trimming expands the model’s ability to discover correct solutions under repeated sampling\. This suggests that TrimSFT does not merely optimize a single deterministic output, but improves the broader solution space explored by the model\. The improvement also extends to best\-of\-8 with majority voting, where TrimSFT remains stronger than or competitive with the baselines across the two models\. Since majority voting requires multiple samples to converge toward the correct answer, these gains indicate that the sampled solutions become more reliably aligned rather than only occasionally correct\. Together with the average@8 results in Table[1](https://arxiv.org/html/2609.09707#S4.T1), these findings show that TrimSFT improves average generation quality, capability coverage, and self\-consistency under repeated sampling\.

### 4\.4Ablation: Margin and Bandwidth

Figure 3:Ablation of marginmmand bandwidthτ\\tauon Qwen2\.5\-Math\-1\.5B\. Left: varyingτ\\tauwith fixedm∈\{1,6\}m\\in\\\{1,6\\\}; right: varyingmmwith fixedτ∈\{1,6\}\\tau\\in\\\{1,6\\\}\. The dashed line denotes the average SFT baseline\.We further study the effect of the two key hyperparameters in TrimSFT: the marginmm, which determines the center of the weighted logit\-gap region, and the bandwidthτ\\tau, which controls how broadly tokens around this margin are emphasized\. Figure[3](https://arxiv.org/html/2609.09707#S4.F3)reports average@8 accuracy, averaged across the five benchmarks, on Qwen2\.5\-Math\-1\.5B under different choices ofmmandτ\\tau, with the full numerical results provided in Appendix[B](https://arxiv.org/html/2609.09707#A2)\.

Figure[3](https://arxiv.org/html/2609.09707#S4.F3)shows that TrimSFT is more sensitive to the bandwidthτ\\tauthan to the marginmm\. Whenmmis fixed, increasingτ\\tauleads to a clear performance drop, especially form=1m=1, indicating that an overly broad weighting function weakens the intended focus on an intermediate logit\-gap region\. In contrast, whenτ=1\\tau=1is fixed, TrimSFT remains consistently strong across different values ofmmand stays well above the SFT baseline, with only moderate variation as the margin changes\. This relative insensitivity tommholds primarily when the bandwidth is small: withτ=6\\tau=6, performance varies more visibly withmm, whereas withτ=1\\tau=1the curve remains comparatively stable\. These results suggest that the width of the selected region is more important than its exact center: TrimSFT benefits most when supervision is concentrated within a relatively narrow band of logit gaps, under which the choice ofmmbecomes less critical\. Additional token\-weight diagnostics further show thatτ\\tauprimarily controls the selectivity and trimming strength of the objective, whilemmmainly shifts the selected logit\-gap region; detailed results are provided in Appendix[C](https://arxiv.org/html/2609.09707#A3)\.

### 4\.5Half\-Trim Variants

We compare TrimSFT with its two half\-trim variants, TrimSFT\-E and TrimSFT\-H, to examine whether the full two\-sided trimming profile is necessary\. TrimSFT\-E trims the hard side only and keeps easy tokens at full weight, whereas TrimSFT\-H trims the easy side only and keeps hard tokens at full weight\. To avoid drawing conclusions from a single margin choice, we evaluate the three variants under representative marginsm∈\{1,3,5\}m\\in\\\{1,3,5\\\}while fixingτ=1\\tau=1\. Table[2](https://arxiv.org/html/2609.09707#S4.T2)reports results on three representative benchmarks\.

Table 2:Comparison between TrimSFT and its half\-trim variants on Qwen2\.5\-Math\-1\.5B with fixedτ=1\\tau=1across representative marginsm∈\{1,3,5\}m\\in\\\{1,3,5\\\}\.Boldmarks the best result among the three variants\.Table[2](https://arxiv.org/html/2609.09707#S4.T2)shows a consistent asymmetry between the two half\-trim variants\. TrimSFT\-E is often competitive with or slightly stronger than TrimSFT on MATH500 and AMC, suggesting that preserving full supervision for higher\-gap tokens can benefit relatively easier or medium\-difficulty benchmarks\. However, this advantage does not consistently transfer to the harder AIME24 benchmark, where TrimSFT achieves the best performance across all three reported margins\. In contrast, TrimSFT\-H performs substantially worse across all settings, mirroring the poor performance of FSFT in Table[1](https://arxiv.org/html/2609.09707#S4.T1)\. These results reveal an asymmetric contribution from the two sides of the weighting profile: suppressing low\-gap tokens appears to be the primary source of stability, whereas trimming high\-gap tokens provides an additional regularization effect\. This helps explain why TrimSFT\-E can remain competitive on easier benchmarks while full TrimSFT provides a more robust trade\-off on harder problems\. Additional gradient\-mass analysis supporting this interpretation is provided in Appendix[E](https://arxiv.org/html/2609.09707#A5)\.

## 5Mechanism: Logit Gap Distribution Analysis

Figure 4:Logit\-gap distributions on 100 randomly sampled training examples for Qwen2\.5\-Math\-1\.5B, aggregated over all response tokens\.To better understand how different fine\-tuning objectives reshape token\-level confidence, i\.e\., the model’s margin\-based confidence in the gold response tokens, we analyze the distribution of logit gaps on the training data\. We randomly sample 100 examples from the training set and compute teacher\-forced logit gaps for every token in the response part of each example\. We then aggregate all token\-level gaps across the sampled examples and plot their kernel density estimates for Base, SFT, DFT, FSFT, and TrimSFT in Figure[4](https://arxiv.org/html/2609.09707#S5.F4)\.

The resulting distributions reveal clear differences in how each objective reshapes token\-level confidence\. Standard SFT shifts the distribution to the right relative to the base model, indicating stronger separation between gold tokens and their competitors after fine\-tuning\. FSFT, in contrast, leaves more mass near small logit gaps, consistent with its emphasis on low\-confidence tokens and its poor performance in Table[1](https://arxiv.org/html/2609.09707#S4.T1)\. DFT also moves the distribution toward larger gaps and appears closer to TrimSFT than to standard SFT, which may help explain why DFT often achieves competitive performance\. However, DFT and TrimSFT reach this behavior through different mechanisms: DFT monotonically emphasizes high\-probability tokens, whereas TrimSFT trims supervision away from both extremes of the logit\-gap spectrum\. As a result, TrimSFT shifts the distribution toward larger gaps while preserving a broad shape and reducing mass in the small\-gap region\. This supports our interpretation that TrimSFT improves reasoning SFT by reshaping token\-level confidence in a more balanced way, rather than simply over\-emphasizing uncertain tokens or uniformly sharpening already confident ones\.

Beyond these distributional analyses, additional token\-category results in Appendix[F](https://arxiv.org/html/2609.09707#A6)further characterize how TrimSFT redistributes the optimization signal, showing a larger share of gradient mass on numeric and mathematical\-symbol tokens and a reduced contribution from generic other tokens\.

## 6Conclusion

We presented TrimSFT, a token\-level reweighting method that uses logit gaps to concentrate supervision on an intermediate confidence region\. By trimming both already well\-separated tokens and tokens with weak current support, TrimSFT provides a non\-monotonic alternative to uniform SFT and monotonic reweighting methods\. Across six base models and five mathematical reasoning benchmarks, TrimSFT consistently improves over standard SFT, while further analyses show gains in capability coverage, self\-consistency, and better trade\-offs than half\-trim variants\. Mechanistically, TrimSFT reshapes token\-level confidence by moving tokens toward more confident regions while avoiding uniform over\-sharpening\. Overall, our results suggest that reasoning SFT can benefit from selectively concentrating supervision within an intermediate logit\-gap region rather than applying uniform pressure across all tokens\.

## Limitations

TrimSFT is evaluated primarily on mathematical reasoning benchmarks with supervised fine\-tuning from base models, so its effectiveness on broader domains such as code generation, open\-ended instruction following, or long\-form reasoning remains to be further studied\. In addition, TrimSFT uses two hyperparameters, the marginmmand bandwidthτ\\tau, to define the emphasized logit\-gap region\. Our ablations show that the method is relatively robust to the exact margin choice and generally improves over standard SFT across a range of settings, suggesting that careful tuning is not necessary to obtain gains\. Still, different model scales or data distributions may benefit from adaptive strategies that adjust the weighting region automatically during training\.

## Ethical Considerations

TrimSFT is a token\-level reweighting method for supervised fine\-tuning and does not involve new data collection or human annotation\. All experiments are conducted using publicly available pretrained models and mathematical reasoning datasets intended for research use\. As with other supervised fine\-tuning methods, improving reasoning capabilities may also strengthen broader generation abilities of language models, which could potentially be misused in downstream applications\.

## Acknowledgments

This research was supported in part by the National Science Foundation under Grant No\. 2452367\.

## References

- AI\-MO \(2024a\)AI\-MOAIMO validation AIME\.Hugging Face\.Note:[https://huggingface\.co/datasets/AI\-MO/aimo\-validation\-aime](https://huggingface.co/datasets/AI-MO/aimo-validation-aime)Cited by:[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px4.p1.1)\.
- AI\-MO \(2024b\)AI\-MOAIMO validation AMC\.Hugging Face\.Note:[https://huggingface\.co/datasets/AI\-MO/aimo\-validation\-amc](https://huggingface.co/datasets/AI-MO/aimo-validation-amc)Cited by:[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px4.p1.1)\.
- Baiet al\.\(2022\)Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Note:Available:[https://arxiv\.org/pdf/2204\.05862](https://arxiv.org/pdf/2204.05862)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p1.1)\.
- Chenet al\.\(2025\)F\. Chen, A\. Raventos, N\. Cheng, S\. Ganguli, and S\. DruckmannRethinking fine\-tuning when scaling test\-time compute: limiting confidence improves mathematical reasoning\.Advances in Neural Information Processing Systems\.Note:Available:[https://papers\.nips\.cc/paper\_files/paper/2025/hash/e8f4eae0a41cab67fdead3aa6b77f083\-Abstract\-Conference\.html](https://papers.nips.cc/paper_files/paper/2025/hash/e8f4eae0a41cab67fdead3aa6b77f083-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p2.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Note:Available:[https://arxiv\.org/pdf/2110\.14168](https://arxiv.org/pdf/2110.14168)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1)\.
- Fanet al\.\(2024\)L\. Fan, W\. Hua, L\. Li, H\. Ling, and Y\. ZhangNPHardEval: dynamic benchmark on reasoning ability of large language models via complexity classes\.InAnnual Meeting of the Association for Computational Linguistics,Note:Available:[https://aclanthology\.org/2024\.acl\-long\.225\.pdf](https://aclanthology.org/2024.acl-long.225.pdf)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p1.1)\.
- Gonget al\.\(2026\)X\. Gong, S\. Wang, H\. Huang, R\. Sun, and S\. LiangVCORE: variance\-controlled optimization\-based reweighting for chain\-of\-thought supervision\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Note:Available:[https://aclanthology\.org/2026\.acl\-long\.1298](https://aclanthology.org/2026.acl-long.1298)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p2.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Note:Available:[https://arxiv\.org/pdf/2407\.21783](https://arxiv.org/pdf/2407.21783)Cited by:[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px2.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InInternational conference on machine learning,Note:Available:[https://arxiv\.org/pdf/1706\.04599](https://arxiv.org/pdf/1706.04599)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px3.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Note:Available:[https://arxiv\.org/pdf/2501\.12948](https://arxiv.org/pdf/2501.12948)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1)\.
- Hanet al\.\(2018\)B\. Han, Q\. Yao, X\. Yu, G\. Niu, M\. Xu, W\. Hu, I\. Tsang, and M\. SugiyamaCo\-teaching: robust training of deep neural networks with extremely noisy labels\.InAdvances in Neural Information Processing Systems,Note:Available:[https://arxiv\.org/pdf/1804\.06872](https://arxiv.org/pdf/1804.06872)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p2.1)\.
- Heet al\.\(2024\)C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.Olympiadbench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.InAnnual Meeting of the Association for Computational Linguistics,Note:Available:[https://aclanthology\.org/2024\.acl\-long\.211\.pdf](https://aclanthology.org/2024.acl-long.211.pdf)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px4.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InConference on Neural Information Processing Systems Datasets and Benchmarks Track,Note:Available:[https://arxiv\.org/pdf/2103\.03874](https://arxiv.org/pdf/2103.03874)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px4.p1.1)\.
- Hugging Face \(2025\)Hugging FaceOpen r1: a fully open reproduction of deepseek\-r1\.External Links:[Link](https://github.com/huggingface/open-r1)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1)\.
- Jiaet al\.\(2025\)Y\. Jia, C\. Zhang, X\. Diao, X\. Yuan, Z\. Ouyang, C\. Ma, and S\. VosoughiWhat makes a good curriculum? disentangling the effects of data ordering on llm mathematical reasoning\.arXiv preprint arXiv:2510\.19099\.Note:Available:[https://arxiv\.org/pdf/2510\.19099](https://arxiv.org/pdf/2510.19099)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p1.1)\.
- Lenget al\.\(2024\)J\. Leng, C\. Huang, B\. Zhu, and J\. HuangTaming overconfidence in llms: reward calibration in rlhf\.arXiv preprint arXiv:2410\.09724\.Note:Available:[https://arxiv\.org/pdf/2410\.09724](https://arxiv.org/pdf/2410.09724)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px3.p1.1)\.
- Lewkowyczet al\.\(2022\)A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo,et al\.Solving quantitative reasoning problems with language models\.InAdvances in Neural Information Processing Systems,Note:Available:[https://arxiv\.org/pdf/2206\.14858](https://arxiv.org/pdf/2206.14858)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px4.p1.1)\.
- Liet al\.\(2024\)J\. Li, E\. Beeching, L\. Tunstall, B\. Lipkin, R\. Soletskyi, S\. Huang, K\. Rasul, L\. Yu, A\. Q\. Jiang, Z\. Shen,et al\.Numinamath: the largest public dataset in ai4maths with 860k pairs of competition math problems and solutions\.Hugging Face repository\.Note:Available:[https://huggingface\.co/collections/AI\-MO/numinamath](https://huggingface.co/collections/AI-MO/numinamath)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px1.p1.1)\.
- Linet al\.\(2026\)J\. Lin, Z\. Wang, K\. Qian, T\. Wang, A\. Srinivasan, H\. Zeng, R\. Jiao, X\. Zhou, J\. Gesi, D\. Wang, Y\. Guo, K\. Zhong, W\. Zhang, sujay sanghavi, C\. Chen, H\. Yun, and L\. LiSFT doesn’t always hurt general capabilities: revisiting domain\-specific fine\-tuning in LLMs\.InInternational Conference on Learning Representations,Note:Available:[https://arxiv\.org/pdf/2509\.20758](https://arxiv.org/pdf/2509.20758)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p2.1)\.
- Linet al\.\(2017\)T\. Lin, P\. Goyal, R\. Girshick, K\. He, and P\. DollárFocal loss for dense object detection\.InInternational Conference on Computer Vision,Note:Available:[https://arxiv\.org/pdf/1708\.02002](https://arxiv.org/pdf/1708.02002)Cited by:[Appendix A](https://arxiv.org/html/2609.09707#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.09707#S1.p2.1),[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2609.09707#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px3.p1.1)\.
- Linet al\.\(2024\)Z\. Lin, Z\. Gou, Y\. Gong, X\. Liu, Y\. Shen, R\. Xu, C\. Lin, Y\. Yang, J\. Jiao, N\. Duan,et al\.Rho\-1: not all tokens are what you need\.arXiv preprint arXiv:2404\.07965\.Note:Available:[https://arxiv\.org/pdf/2404\.07965](https://arxiv.org/pdf/2404.07965)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2022\)B\. Liu, I\. Ben Ayed, A\. Galdran, and J\. DolzThe devil is in the margin: margin\-based label smoothing for network calibration\.InConference on Computer Vision and Pattern Recognition,Note:Available:[https://arxiv\.org/pdf/2111\.15430](https://arxiv.org/pdf/2111.15430)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2025\)C\. Liu, Y\. Yuan, Y\. Yin, Y\. Xu, X\. Xu, Z\. Chen, Y\. Wang, L\. Shang, Q\. Liu, and M\. ZhangSafe: enhancing mathematical reasoning in large language models via retrospective step\-aware formal verification\.InAnnual Meeting of the Association for Computational Linguistics,Note:Available:[https://aclanthology\.org/2025\.acl\-long\.594\.pdf](https://aclanthology.org/2025.acl-long.594.pdf)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p1.1)\.
- Luoet al\.\(2025\)H\. Luo, Q\. Sun, C\. Xu, P\. Zhao, J\. Lou, C\. Tao, X\. Geng, Q\. Lin, S\. Chen, Y\. Tang, and D\. ZhangWizardMath: empowering mathematical reasoning for large language models via reinforced evol\-instruct\.InInternational Conference on Learning Representations,Note:Available:[https://arxiv\.org/pdf/2308\.09583](https://arxiv.org/pdf/2308.09583)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1)\.
- Menget al\.\(2024\)Y\. Meng, M\. Xia, and D\. ChenSimpo: simple preference optimization with a reference\-free reward\.InAdvances in Neural Information Processing Systems,Note:Available:[https://arxiv\.org/pdf/2405\.14734](https://arxiv.org/pdf/2405.14734)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px3.p1.1)\.
- Meta AI \(2024\)Meta AILlama 3\.2: revolutionizing edge AI and vision with open, customizable models\.Note:[https://ai\.meta\.com/blog/llama\-3\-2\-connect\-2024\-vision\-edge\-mobile\-devices/](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)Meta AI Blog, accessed 2026Cited by:[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px2.p1.1)\.
- Mülleret al\.\(2019\)R\. Müller, S\. Kornblith, and G\. E\. HintonWhen does label smoothing help?\.InAdvances in Neural Information Processing Systems,Note:Available:[https://arxiv\.org/pdf/1906\.02629](https://arxiv.org/pdf/1906.02629)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px3.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Note:Available:[https://arxiv\.org/pdf/2203\.02155](https://arxiv.org/pdf/2203.02155)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p1.1)\.
- Pereyraet al\.\(2017\)G\. Pereyra, G\. Tucker, J\. Chorowski, Ł\. Kaiser, and G\. HintonRegularizing neural networks by penalizing confident output distributions\.arXiv preprint arXiv:1701\.06548\.Note:Available:[https://arxiv\.org/pdf/1701\.06548](https://arxiv.org/pdf/1701.06548)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p2.1),[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px3.p1.1)\.
- Ruanet al\.\(2025\)Z\. Ruan, Y\. Li, H\. Zhu, Y\. Chen, P\. Li, Y\. Liu, and G\. ChenEnhancing large language model reasoning via selective critical token fine\-tuning\.arXiv preprint arXiv:2510\.10974\.Note:Available:[https://arxiv\.org/pdf/2510\.10974](https://arxiv.org/pdf/2510.10974)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px2.p1.1)\.
- Saxtonet al\.\(2019\)D\. Saxton, E\. Grefenstette, F\. Hill, and P\. KohliAnalysing mathematical reasoning abilities of neural models\.arXiv preprint arXiv:1904\.01557\.Note:Available:[https://arxiv\.org/pdf/1904\.01557](https://arxiv.org/pdf/1904.01557)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Note:Available:[https://arxiv\.org/pdf/2402\.03300](https://arxiv.org/pdf/2402.03300)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px2.p1.1)\.
- Toshniwalet al\.\(2024\)S\. Toshniwal, I\. Moshkov, S\. Narenthiran, D\. Gitman, F\. Jia, and I\. GitmanOpenmathinstruct\-1: a 1\.8 million math instruction tuning dataset\.InAdvances in Neural Information Processing Systems,Note:Available:[https://arxiv\.org/pdf/2402\.10176](https://arxiv.org/pdf/2402.10176)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.Mmlu\-pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems,Note:Available:[https://arxiv\.org/pdf/2406\.01574](https://arxiv.org/pdf/2406.01574)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p1.1)\.
- Weiet al\.\(2022a\)H\. Wei, R\. Xie, H\. Cheng, L\. Feng, B\. An, and Y\. LiMitigating neural network overconfidence with logit normalization\.InInternational Conference on Machine Learning,Note:Available:[https://arxiv\.org/pdf/2205\.09310](https://arxiv.org/pdf/2205.09310)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p2.1),[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px3.p1.1)\.
- Weiet al\.\(2022b\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Note:Available:[https://arxiv\.org/pdf/2201\.11903](https://arxiv.org/pdf/2201.11903)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p1.1)\.
- Wuet al\.\(2026\)Y\. Wu, Y\. Zhou, Z\. Ziheng, Y\. Peng, X\. Ye, X\. Hu, W\. Zhu, L\. Qi, M\. Yang, and X\. YangOn the generalization of SFT: a reinforcement learning perspective with reward rectification\.InInternational Conference on Learning Representations,Note:Available:[https://arxiv\.org/pdf/2508\.05629](https://arxiv.org/pdf/2508.05629)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p2.1),[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2609.09707#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px3.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Note:Available:[https://arxiv\.org/pdf/2505\.09388](https://arxiv.org/pdf/2505.09388)Cited by:[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px2.p1.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin,et al\.Qwen2\.5\-Math technical report: toward mathematical expert model via self\-improvement\.arXiv preprint arXiv:2409\.12122\.Note:Available:[https://arxiv\.org/pdf/2409\.12122](https://arxiv.org/pdf/2409.12122)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.09707#S4.SS1.SSS0.Px2.p1.1)\.
- Yeet al\.\(2025\)Y\. Ye, Z\. Huang, Y\. Xiao, E\. Chern, S\. Xia, and P\. LiuLimo: less is more for reasoning\.arXiv preprint arXiv:2502\.03387\.Note:Available:[https://arxiv\.org/pdf/2502\.03387](https://arxiv.org/pdf/2502.03387)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2024\)L\. Yu, W\. Jiang, H\. Shi, J\. YU, Z\. Liu, Y\. Zhang, J\. Kwok, Z\. Li, A\. Weller, and W\. LiuMetaMath: bootstrap your own mathematical questions for large language models\.InInternational Conference on Learning Representations,Note:Available:[https://arxiv\.org/pdf/2309\.12284](https://arxiv.org/pdf/2309.12284)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p1.1),[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1)\.
- Yueet al\.\(2024\)X\. Yue, X\. Qu, G\. Zhang, Y\. Fu, W\. Huang, H\. Sun, Y\. Su, and W\. ChenMAmmoTH: building math generalist models through hybrid instruction tuning\.InInternational Conference on Learning Representations,Note:Available:[https://arxiv\.org/pdf/2309\.05653](https://arxiv.org/pdf/2309.05653)Cited by:[§1](https://arxiv.org/html/2609.09707#S1.p1.1),[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026\)H\. Zhang, B\. Zhang, M\. Cheng, S\. Hassanpour, W\. Ma, and S\. VosoughiWhen linguistic and internal confidence diverge in large language models\.arXiv preprint arXiv:2608\.28382\.Note:Available:[https://arxiv\.org/pdf/2608\.28382](https://arxiv.org/pdf/2608.28382)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2024\)M\. Zhang, M\. Huang, R\. Shi, L\. Guo, C\. Peng, P\. Yan, Y\. Zhou, and X\. QiuCalibrating the confidence of large language models by eliciting fidelity\.InConference on Empirical Methods in Natural Language Processing,Note:Available:[https://aclanthology\.org/2024\.emnlp\-main\.173\.pdf](https://aclanthology.org/2024.emnlp-main.173.pdf)Cited by:[§2](https://arxiv.org/html/2609.09707#S2.SS0.SSS0.Px3.p1.1)\.

## Contents of Appendix

## Appendix AExperiment Setting

#### Parameter setting\.

We train all models for one epoch using the AdamW optimizer with a learning rate of5×10−55\\times 10^\{\-5\}, cosine learning rate decay with warmup, and gradient clipping\. Training is conducted with bfloat16 mixed precision and FlashAttention\-2 acceleration under the FSDP framework\. Unless otherwise specified, the global batch size is 24 with a per\-device micro\-batch size of 4, and the maximum sequence length is set to 2048\. For TrimSFT, we set the margin tom=1\.5m=1\.5and the bandwidth toτ=0\.8\\tau=0\.8\.

#### Baseline setting\.

All four methods, SFT, FSFT, DFT, and TrimSFT, are trained under an identical configuration, including optimizer, learning rate, schedule, batch size, sequence length, precision, and number of epochs\. They differ only in the per\-token weightwtw\_\{t\}applied to the cross\-entropy loss, as summarized in Section[3\.3](https://arxiv.org/html/2609.09707#S3.SS3)\. The only method\-specific hyperparameters beyond this shared recipe are the marginmmand bandwidthτ\\tauin TrimSFT, and the focusing parameterγ\\gammain FSFT, for which we useγ=2\\gamma=2following common practice in the focal\-loss literature\([Lin et al\., 2017](https://arxiv.org/html/2609.09707#bib.bib17)\)\. DFT introduces no additional hyperparameters\.

#### Evaluation protocol\.

For evaluation, we sample eight generations per problem using temperature1\.01\.0, top\-ppsampling withp=1\.0p=1\.0, and a maximum generation length of 2048 tokens\. We use temperature1\.01\.0to encourage diverse sampled solutions, which is necessary for evaluating average@8, pass@8, and best\-of\-8 under repeated sampling\. Average@8 is computed as the mean correctness over the eight sampled generations, pass@8 checks whether at least one generation is correct, and best\-of\-8 uses majority voting over the extracted final answers\. We extract final answers from the generated solutions and judge correctness using the same evaluation protocol across all methods and benchmarks\.

#### Computing infrastructure\.

Experiments are conducted on NVIDIA RTX A6000 GPUs, each with 48GB memory, using CUDA 12\.8 and NVIDIA driver 570\.207\. Each training run uses two GPUs with FSDP unless otherwise specified\.

## Appendix BDetailed Results for Ablations

Table[3](https://arxiv.org/html/2609.09707#A2.T3)and Table[4](https://arxiv.org/html/2609.09707#A2.T4)report the full per\-benchmark results for the margin and bandwidth ablation study shown in Figure[3](https://arxiv.org/html/2609.09707#S4.F3)\. Each setting is denoted by\(m,τ\)\(m,\\tau\), wheremmis the margin andτ\\tauis the bandwidth\. While the main text summarizes the averaged trends across the evaluated benchmarks, this appendix provides the complete per\-benchmark results for both varying the bandwidth under fixed margins and varying the margin under fixed bandwidths\.

Table 3:Ablation results with fixedmmand varyingτ\\tau\. Each setting is denoted as\(m,τ\)\(m,\\tau\)\.Table 4:Ablation results with fixedτ\\tauand varyingmm\. Each setting is denoted as\(m,τ\)\(m,\\tau\)\.The per\-benchmark results are consistent with the averaged trends in the main text\. When the marginmmis fixed, increasing the bandwidthτ\\taugenerally reduces performance, especially whenτ\\taubecomes large\. In contrast, whenτ=1\\tau=1is fixed, the method remains stronger than standard SFT across a wide range of margin values, indicating that it does not require careful tuning of the exact margin location to obtain improvements\.

## Appendix CUnderstanding the Roles ofmmandτ\\tau

To further explain the different sensitivities of TrimSFT to the bandwidthτ\\tauand the marginmm, we conduct a token\-weight analysis under the same experimental setting as Section[4\.4](https://arxiv.org/html/2609.09707#S4.SS4), using Qwen2\.5\-Math\-1\.5B\. Specifically, we compute logit gaps on 40,082 gold response tokens sampled from the training data and derive the corresponding TrimSFT weights:

wt=exp⁡\(−\(Δt−m\)22​τ2\),w\_\{t\}=\\exp\\\!\\left\(\-\\frac\{\(\\Delta\_\{t\}\-m\)^\{2\}\}\{2\\tau^\{2\}\}\\right\),\(9\)whereΔt=zt,yt−maxv≠yt⁡zt,v\\Delta\_\{t\}=z\_\{t,y\_\{t\}\}\-\\max\_\{v\\neq y\_\{t\}\}z\_\{t,v\}\.

We report two diagnostics\.High\-Weight \(%\)is the fraction of tokens withwt\>0\.5w\_\{t\}\>0\.5, while theEffective Token Ratio, denoted asReffR\_\{\\mathrm\{eff\}\}, is defined as

Reff=\(∑t=1Nwt\)2N​∑t=1Nwt2\.R\_\{\\mathrm\{eff\}\}=\\frac\{\\left\(\\sum\_\{t=1\}^\{N\}w\_\{t\}\\right\)^\{2\}\}\{N\\sum\_\{t=1\}^\{N\}w\_\{t\}^\{2\}\}\.\(10\)whereNNis the number of response tokens\. A largerReffR\_\{\\mathrm\{eff\}\}indicates that supervision is distributed more uniformly across tokens, with uniform SFT corresponding toReff=1R\_\{\\mathrm\{eff\}\}=1\.

Table 5:Effect of margin and bandwidth on TrimSFT token weighting for Qwen2\.5\-Math\-1\.5B\. Avg\. denotes average@8 accuracy across the five mathematical reasoning benchmarks\.Table[5](https://arxiv.org/html/2609.09707#A3.T5)shows that varyingmmfrom 1 to 6 withτ=1\\tau=1leaves the fraction of high\-weight tokens nearly unchanged, while increasingτ\\tausubstantially broadens the weighted region and movesReffR\_\{\\mathrm\{eff\}\}toward uniform SFT\. These results provide additional evidence thatmmmainly shifts the location of the weighting band, whereasτ\\taucontrols its selectivity and trimming strength\.

## Appendix DHalf\-Trim Variants

Table[6](https://arxiv.org/html/2609.09707#A4.T6)reports additional results for TrimSFT and its two half\-trim variants, TrimSFT\-E and TrimSFT\-H, on Qwen2\.5\-Math\-1\.5B\. The main text reports representative marginsm∈\{1,3,5\}m\\in\\\{1,3,5\\\}, while this appendix includes the full set of marginsm∈\{1,2,3,4,5\}m\\in\\\{1,2,3,4,5\\\}\. All variants are evaluated with fixedτ=1\\tau=1\.

Table 6:Comparison between TrimSFT and its half\-trim variants with fixedτ=1\\tau=1\.Boldmarks the best result among the three variants\.The results are consistent with the trends discussed in the main text\. TrimSFT\-E is often competitive with TrimSFT on relatively easier or medium\-difficulty benchmarks such as MATH500 and AMC, suggesting that preserving full supervision on high\-gap tokens can sometimes be useful\. However, TrimSFT achieves better results on the harder AIME24 benchmark across the reported margins\. In contrast, TrimSFT\-H consistently performs much worse than both TrimSFT and TrimSFT\-E, indicating that preserving full supervision on low\-gap tokens while trimming high\-gap tokens leads to an inferior trade\-off\. Overall, these results support the benefit of two\-sided trimming over relying on either half\-trim variant alone\.

## Appendix EGradient\-Mass Analysis

To further examine the asymmetric behavior of the half\-trim variants, we analyze how each weighting scheme redistributes the token\-level optimization signal\. We conduct this analysis on 100 randomly sampled training examples from NuminaMath\-CoT, covering 40,082 response tokens, using Qwen2\.5\-Math\-1\.5B\. Following the setting in Section[4\.5](https://arxiv.org/html/2609.09707#S4.SS5), we fixτ=1\\tau=1and considerm∈\{1,3,5\}m\\in\\\{1,3,5\\\}\. For each response token, we compute the weighted logit\-gradient magnitude

gt=wt​‖softmax⁡\(zt\)−onehot⁡\(yt\)‖2,g\_\{t\}=w\_\{t\}\\left\\\|\\operatorname\{softmax\}\(z\_\{t\}\)\-\\operatorname\{onehot\}\(y\_\{t\}\)\\right\\\|\_\{2\},\(11\)and report the fraction of total gradient mass assigned to different logit\-gap regions\. For each\(m,τ\)\(m,\\tau\)setting, we define the low\-, middle\-, and high\-gap regions asΔt<m−τ\\Delta\_\{t\}<m\-\\tau,m−τ≤Δt≤m\+τm\-\\tau\\leq\\Delta\_\{t\}\\leq m\+\\tau, andΔt\>m\+τ\\Delta\_\{t\}\>m\+\\tau, respectively\.

Table 7:Distribution of weighted gradient mass \(%\) across logit\-gap regions for TrimSFT and its half\-trim variants on Qwen2\.5\-Math\-1\.5B\. All settings useτ=1\\tau=1\.Table[7](https://arxiv.org/html/2609.09707#A5.T7)reveals a clear asymmetry between the two sides of the weighting profile\. TrimSFT\-H, which preserves full supervision on low\-gap tokens, concentrates most of its gradient mass in the low\-gap region, reaching 71\.78%, 93\.80%, and 99\.04% asmmincreases\. In contrast, both TrimSFT and TrimSFT\-E substantially suppress this region and maintain a larger share of the optimization signal in the intermediate region\. Full TrimSFT further reduces the contribution of high\-gap tokens relative to TrimSFT\-E, providing an additional regularization effect on already well\-separated tokens\. These results support the interpretation that low\-gap trimming is the primary source of stability, while high\-gap trimming provides a complementary regularization effect\.

## Appendix FToken\-Category Analysis

Following the same experimental setting as Section[E](https://arxiv.org/html/2609.09707#A5), we conduct the token\-category analysis on 100 randomly sampled training examples from NuminaMath\-CoT using Qwen2\.5\-Math\-1\.5B, covering 40,082 response tokens\. We group the decoded response tokens into four lightweight categories and compare standard SFT with TrimSFT under\(m,τ\)=\(1,1\)\(m,\\tau\)=\(1,1\)\.

Table 8:Token\-category analysis on Qwen2\.5\-Math\-1\.5B\. Num\., MS, and RC denote numeric tokens, mathematical symbols, and reasoning connectors, respectively\. Ratio is the token fraction, Weight is the average TrimSFT weight, Mid\. is the intermediate\-gap fraction, and Grad\. is the gradient\-mass fraction\.Table[8](https://arxiv.org/html/2609.09707#A6.T8)shows that TrimSFT assigns a larger share of gradient mass to numeric and mathematical\-symbol tokens than standard SFT, while reducing the contribution of the broadOthercategory\. In particular, the gradient\-mass share increases from 5\.55% to 10\.31% for numeric tokens and from 15\.76% to 22\.78% for mathematical symbols\. These results suggest that the reweighted optimization signal remains closely associated with math\-relevant token categories rather than being concentrated primarily on generic response tokens\.