DEL: Digit Entropy Loss for Numerical Learning of Large Language Models

# 1 Introduction
Source: [https://arxiv.org/html/2605.20369](https://arxiv.org/html/2605.20369)
![[Uncaptioned image]](https://arxiv.org/html/2605.20369v1/polyu-logo.png)![[Uncaptioned image]](https://arxiv.org/html/2605.20369v1/vclab-logo.jpg)VISUAL COMPUTING LAB

POLYU VCLAB • PREPRINT 2026

DEL: Digit Entropy Loss for Numerical Learning of Large Language Models

Zhaohui Zheng1Chenhang He1Shihao Wang1Yuxuan Li2Ming\-Ming Cheng2Lei Zhang†\\dagger1

1The Hong Kong Polytechnic University2VCIP, College of Computer Science, Nankai University

Abstract\.Number prediction stands as a fundamental capability of large language models \(LLMs\) in mathematical problem\-solving and code generation\. The widely adopted maximum likelihood estimation \(MLE\) for LLM training is not tailored to number prediction\. Recently, penalty\-driven approaches, e\.g\., Number Token Loss and Discretized Distance Loss, introduce an inductive bias of numerical distance but induce over\-sharpened and over\-flattened digit distributions, respectively\. In this paper, we make an in\-depth analysis on LLM numerical learning, and show that existing numerical learning methods conceptually follow a criterion\-distance formulation, where the criterion term represents optimization pattern and the distance term instills geometric prior\. Consequently, we presentDigitEntropyLoss \(DEL\) for auto\-regressive numerical learning, which reformulates the conventional unsupervised entropy optimization in three key designs: leveraging digit conditional probability and binary cross\-entropy to guide the entropy optimization into a supervised manner; deprecating the distance term to bypass the issue of numerical distance; and generalizing the integer\-based numerical learning to floating\-point number optimization, enabling more accurate number prediction\. Our DEL formulation can incorporate integers, decimals, and decimal points, expanding the learning objective from a single digit to the floating\-point number domain\. Experiments conducted on seven mathematical reasoning benchmarks with four representative LLMs, including CodeLlama, Mistral, DeepSeek, and Qwen\-2\.5, demonstrate that DEL consistently outperforms its counterparts in both overall prediction accuracy and numerical distance\.

KEYWORDS :Large Language Models, Numerical Learning, Loss function, PolyU VCLab

Large language models \(LLMs\) have demonstrated remarkable natural language processing capabilities, gradually evolving from the handling of simple human\-computer dialogues to complex advanced reasoning, such as scientific computation\[[58](https://arxiv.org/html/2605.20369#bib.bib3),[52](https://arxiv.org/html/2605.20369#bib.bib5),[49](https://arxiv.org/html/2605.20369#bib.bib4)\]and code generation\[[44](https://arxiv.org/html/2605.20369#bib.bib15),[66](https://arxiv.org/html/2605.20369#bib.bib1),[53](https://arxiv.org/html/2605.20369#bib.bib2)\]\. As numbers are ubiquitous used in our daily life, the ability to understand, reason with and accurately generate numerical information has emerged as a foundational capability of LLM, serving as the bedrock for quantitative analysis, logical deduction, and the stability of executing algorithms within the aforementioned domains\.

Since the pioneering work in\[[6](https://arxiv.org/html/2605.20369#bib.bib49)\], a common approach to optimize language models is maximum likelihood estimation \(MLE\), which maximizes the likelihood of conditional probabilities of the generated language sequence\. However, it has been found that optimizing LLMs using MLE is sub\-optimal as the zero\-avoiding property of MLE forces the model to assign non\-trivial probabilities to any observed tokens in the training data\. This will lead to issues such as text degeneration\[[21](https://arxiv.org/html/2605.20369#bib.bib65)\], deviation from human\-like text\[[68](https://arxiv.org/html/2605.20369#bib.bib38)\], and data void overestimation\[[22](https://arxiv.org/html/2605.20369#bib.bib55)\]\. To alleviate such problems, later representative methods\[[22](https://arxiv.org/html/2605.20369#bib.bib55),[68](https://arxiv.org/html/2605.20369#bib.bib38),[43](https://arxiv.org/html/2605.20369#bib.bib39)\]have considered the use of new distance alternatives, which show effectiveness in the prediction of general texts\. However, their improvements in number prediction remain limited because these approaches neglect the characteristics of numerical optimization\.

Recent advances in numerical learning\[[67](https://arxiv.org/html/2605.20369#bib.bib9),[10](https://arxiv.org/html/2605.20369#bib.bib36)\]offer a promising alternative by introducing a penalty\-driven approach\. By adding a penalty term to the cross entropy loss to measure the proximity between digits, the supervision signals carried by digits can provide positive guidance to LLMs\. As shown in Fig\.[1](https://arxiv.org/html/2605.20369#S1.F1), NTL\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]explicitly suppresses the probability of non\-target tokens to nearly 0, while DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]enforces a smooth objective distribution\. As we will analyze later, both NTL and DIST2Loss conceptually follow a criterion\-distance formulation, where the criterion term provides the optimization pattern for the LLMs to learn and the distance term determines the degree of penalty imposed on each digit probability\. However, these two losses have some limitations\.Firstly, they heavily rely on prior knowledge, i\.e\., the definition of the distance term\.For example, if the target digit is 3, predicting 7 is further away than predicting 4, and therefore a greater penalty will be imposed\. A rigorous constraint as in\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]can easily lead to over\-sharpened digit distribution, while a soft constraint as in\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]can produce greater uncertainty in generating digits\. In addition, the semantic information of numbers is multifaceted and cannot be fully captured by numerical distance alone\.Secondly, they lack holistic number awareness\.Specifically, NTL\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]operates within a single\-digit optimization paradigm, which limits its ability to model the global numerical structure of multi\-digit numbers\. DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]introduces the digital place weighting for integers, but the discontinuous penalty between the integers and the decimals still persists since the decimal is treated as a separate integer\.

\\begin\{overpic\}\[width=433\.62pt\]\{pic/optimization\-goal\} \\put\(11\.0,17\.8\)\{MLE\} \\put\(34\.5,17\.8\)\{NTL \\cite\[cite\]\{\[\\@@bibref\{Number\}\{NTL\}\{\}\{\}\]\}\} \\put\(57\.2,17\.8\)\{DIST$\{\}^\{2\}$Loss \\cite\[cite\]\{\[\\@@bibref\{Number\}\{DIST2loss\}\{\}\{\}\]\}\} \\put\(87\.0,17\.8\)\{Ours\} \\end\{overpic\}Figure 1:Comparison of the optimization objectives of MLE, NTL\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\], DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\], and our proposed DEL\. In the illustration, 5 is the target token\.In this paper, we proposeDigitEntropyLoss \(DEL\) for auto\-regressive numerical learning\. Inspired by the phenomenon revealed in\[[60](https://arxiv.org/html/2605.20369#bib.bib56)\], high entropy tokens are mostly logic connectors and transitions, while low entropy tokens exhibit determinacy\. We argue that highly deterministic generation is fundamentally crucial for numerical tokens in language models\. This is because as a discrete and precise symbolic system, the semantics of numbers come from their magnitude, digit sequence, and proportional relationships\. Moreover, numerical determinacy serves as an indispensable prerequisite for reliable LLM deployment in downstream tasks\. Guided by these insights, we shift our attention towards entropy minimization\. However, conventional entropy minimization may struggle in multiple local optima because any one\-hot vector is one of the solutions\. To address this issue, we reformulate the conventional entropy optimization with the following three key designs\. First, we propose to use the combination of digit conditional probability and binary cross\-entropy as the new criterion term, where we particularly concentrate on the subspace of digits\. Second, we deprecate the distance term because it does not strictly follow the semantic representations of digits learned by LLMs from massive real\-world corpora\. Finally, we generalize the numerical learning to floating\-point number optimization, which recognizes all floating\-point numbers in the ground\-truth token sequences during training, enabling the language models with holistic number awareness\.

To evaluate our proposed method, we apply DEL to several popular LLMs, including CodeLlama, Mistral, DeepSeek\-Math, and Qwen\-2\.5\. We demonstrate that DEL is capable of boosting number prediction performance in both Chain\-of\-Thought \(CoT\) reasoning and Program\-of\-Thought \(PoT\) code generation\. The experiments on seven mathematical reasoning benchmarks show that DEL achieves consistent accuracy gains and lower numerical distance over its state\-of\-the\-art counterparts\.

## 2 Related Work

### 2\.1Probabilistic Language Models

Probabilistic language models have long been used in Natural Language Processing \(NLP\), e\.g\., neural machine translation\[[37](https://arxiv.org/html/2605.20369#bib.bib43),[45](https://arxiv.org/html/2605.20369#bib.bib44)\], speech recognition\[[15](https://arxiv.org/html/2605.20369#bib.bib45),[40](https://arxiv.org/html/2605.20369#bib.bib46)\], and optical character recognition \(OCR\)\[[14](https://arxiv.org/html/2605.20369#bib.bib47),[26](https://arxiv.org/html/2605.20369#bib.bib48)\], etc\. This language modeling approach can be dated back to pioneer work\[[6](https://arxiv.org/html/2605.20369#bib.bib49)\]in 1992\. By modeling a series of conditional probabilities over the sequences of symbols pertaining to a language, researchers have witnessed extraordinary reasoning capability emerging from the language models after training on large\-scale humanity data\. Then, the emergence of LSTM\[[20](https://arxiv.org/html/2605.20369#bib.bib50)\]and transformers\[[57](https://arxiv.org/html/2605.20369#bib.bib51)\]greatly enhanced the ability of language models to model long sequences\. Subsequently, word embedding techniques have evolved from learning static representations for words\[[4](https://arxiv.org/html/2605.20369#bib.bib52),[35](https://arxiv.org/html/2605.20369#bib.bib53)\]to context\-aware, dynamic representations like BERT\[[12](https://arxiv.org/html/2605.20369#bib.bib54)\]\. Another interesting approach is prompting method\. With some task\-specific exemplars, language models\[[7](https://arxiv.org/html/2605.20369#bib.bib25)\]can better perform a variety of tasks and even generate the chain\-of\-thought \(CoT\)\[[61](https://arxiv.org/html/2605.20369#bib.bib61),[36](https://arxiv.org/html/2605.20369#bib.bib34),[63](https://arxiv.org/html/2605.20369#bib.bib62)\], which shows better performance in math word problems\. Recently, Program\-of\-Thought \(PoT\)\[[8](https://arxiv.org/html/2605.20369#bib.bib63)\]and Program\-aided Language Models \(PAL\)\[[13](https://arxiv.org/html/2605.20369#bib.bib64)\]have explored code generation to overcome the numerical calculation errors and the inefficiency of multiple iterations\.

### 2\.2Probabilistic Sequence Optimization

When generating a sequence, the language model performs next token prediction in an autoregressive manner\. After gathering the inference results, the maximum likelihood estimation is used to optimize the sequence probabilities\. The idea of improving such sequence probability is not new\. For example, MixCE\[[68](https://arxiv.org/html/2605.20369#bib.bib38)\]combines the reversed cross\-entropy to help the language model to generate more like humans\. TaiLr\[[22](https://arxiv.org/html/2605.20369#bib.bib55)\]utilizes the total variation distance to reduce the penalization intensity of low probability predictions\. Another attempt is EMO\[[43](https://arxiv.org/html/2605.20369#bib.bib39)\], which considers the Earth Mover Distance \(EMD\) to drive the differentiable upper bound of EMD\. Recently, optimization for numerical sequences has received increasing attention, as predicting numerical sequences is the core of many reasoning tasks, e\.g\., arithmetic\[[65](https://arxiv.org/html/2605.20369#bib.bib57)\], mathematical reasoning\[[11](https://arxiv.org/html/2605.20369#bib.bib77),[19](https://arxiv.org/html/2605.20369#bib.bib78)\], code generation\[[44](https://arxiv.org/html/2605.20369#bib.bib15),[66](https://arxiv.org/html/2605.20369#bib.bib1)\], etc\. It is also a fundamental problem in some vision language models \(VLM\)\[[31](https://arxiv.org/html/2605.20369#bib.bib58),[9](https://arxiv.org/html/2605.20369#bib.bib59),[59](https://arxiv.org/html/2605.20369#bib.bib60)\], where the language model begins to take on some object\-related visual grounding and clue inferring\. To optimize the numerical sequence, a successful approach, namely Number Token Loss \(NTL\)\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\], has been found useful to mathematical reasoning\. NTL incorporates digit distance into the objective function, thereby endowing the language models with the awareness of digit prediction errors\. More recently, DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]proposes to mimic a digit\-distance\-driven soft target, where a KL\-divergence is imposed between the predict sequence probabilities and the soft targets\. Despite noticeable improvement over the maximum likelihood estimation, they are essentially based on a heuristic definition of numerical distance\. Motivated by the aforementioned methods, we study the numerical learning in language models with in\-depth analyses on the existing approaches, exploring effective means to adapt the numerical nature\.

## 3 Numerical Learning in Language Models

Background\. In reasoning tasks that involve numeric outputs, the language model𝐌\\mathbf\{M\}with parametersΘ\\Thetatypically predicts numbers in an auto\-regressive, token\-by\-token manner\[[42](https://arxiv.org/html/2605.20369#bib.bib12),[41](https://arxiv.org/html/2605.20369#bib.bib24),[54](https://arxiv.org/html/2605.20369#bib.bib13)\]\. LetXXbe a language sequence of lengthTT\. When predicting thett\-th token, the language model outputs logitszt=𝐌Θ\(⋅\|x<t\)z\_\{t\}=\\mathbf\{M\}\_\{\\Theta\}\(\\cdot\|x\_\{<t\}\), and converts it to the conditional probability distribution𝐩\(⋅\|x<t\)=𝒮\(zt\)\\mathbf\{p\}\(\\cdot\|x\_\{<t\}\)=\\mathcal\{S\}\(z\_\{t\}\)over the vocabularyVV, where𝒮​\(⋅\)\\mathcal\{S\}\(\\cdot\)is the Softmax function\. We call the probability𝐩\(⋅\|x<t\)\\mathbf\{p\}\(\\cdot\|x\_\{<t\}\)the full space probabilitysince it represents the probability of takingxtx\_\{t\}as the predicted token over the full vocabulary token spaceVV\. During training, all the probabilities are gathered to a large probability sequence𝐩\\mathbf\{p\}\. Here, we only discuss the digit tokens and omit the non\-digital tokens for brevity\. Then, the standard MLE is used to minimize the distance between the probability sequence𝐩\\mathbf\{p\}and the ground\-truth digit token sequence𝐠\\mathbf\{g\}\. This solution, also known as the language modeling loss, has been dominantly adopted in various popular language models, including GPT\[[41](https://arxiv.org/html/2605.20369#bib.bib24),[7](https://arxiv.org/html/2605.20369#bib.bib25),[1](https://arxiv.org/html/2605.20369#bib.bib26)\], Llama\[[54](https://arxiv.org/html/2605.20369#bib.bib13),[55](https://arxiv.org/html/2605.20369#bib.bib14),[44](https://arxiv.org/html/2605.20369#bib.bib15)\], Qwen\[[3](https://arxiv.org/html/2605.20369#bib.bib27),[51](https://arxiv.org/html/2605.20369#bib.bib28),[39](https://arxiv.org/html/2605.20369#bib.bib29)\], Mistral\[[23](https://arxiv.org/html/2605.20369#bib.bib23)\], Vicuna\[[69](https://arxiv.org/html/2605.20369#bib.bib11)\], etc\.

To facilitate numerical learning, the NTL method\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]adds a penalty term focusing on the digits during training, which can be written as:

ℒN​T​L=ℋ​\(𝐩,𝐠\)⏟cross\-entropy\+α∑𝐠t∈Vd𝐩^\(⋅\|x<t\)\|\|Vd−𝐠t\|\|1,\\mathcal\{L\}\_\{NTL\}=\\underbrace\{\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)\}\_\{\\text\{cross\-entropy\}\}\+\\alpha\\sum\\limits\_\{\\mathbf\{g\}\_\{t\}\\in V\_\{d\}\}\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\|\|V\_\{d\}\-\\mathbf\{g\}\_\{t\}\|\|\_\{1\},\(1\)whereVd=\{0,1,⋯,9\}V\_\{d\}=\\\{0,1,\\cdots,9\\\}is the digit vocabulary,𝐩^\(⋅\|x<t\)=𝒮\(z^t\)=𝒮\(𝐌Θ\(⋅\|x<t\)\|Vd\)\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)=\\mathcal\{S\}\(\\hat\{z\}\_\{t\}\)=\\mathcal\{S\}\(\\mathbf\{M\}\_\{\\Theta\}\(\\cdot\|x\_\{<t\}\)\|\_\{V\_\{d\}\}\)denotesthe digit conditional probabilityoverVdV\_\{d\},𝐠t\\mathbf\{g\}\_\{t\}denotes thett\-th ground\-truth token, andα\\alphais a hyperparameter balancing the two terms\. Due to the introduction of penalty term, the predicted digit probabilities will shift towards the target digit distribution\. Note that we use the NTL with the Wasserstein\-1 Distance \(NTL\-WAS\) in the above equation since it generally achieves better accuracy as evidenced by\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]both theoretically and experimentally\.

Another recent study DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]suggests distance modeling in loss functions\. It first formulates a target distribution𝐪t\\mathbf\{q\}\_\{t\}:

𝐪t=𝒮​\(−ρ​\(Vd,𝐠t\),τ\),𝐠t∈Vd,\\mathbf\{q\}\_\{t\}=\\mathcal\{S\}\(\-\\rho\(V\_\{d\},\\mathbf\{g\}\_\{t\}\),\\tau\),\\quad\\mathbf\{g\}\_\{t\}\\in V\_\{d\},\(2\)where𝒮​\(⋅,τ\)\\mathcal\{S\}\(\\cdot,\\tau\)denotes the generalized softmax function with temperatureτ\\tau,ρ\\rhois a pre\-defined distance metric\. Then, a KL\-divergence loss is imposed between the predicted digit probability𝐩^\(⋅\|x<t\)\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)and the𝐪t\\mathbf\{q\}\_\{t\}defined above, which can be written as:

ℒd​i​s​t=ℋ\(𝐩,𝐠\)\+α∑𝐠t∈VdℒKL\(𝐪t∥𝐩^\(⋅\|x<t\)\)\.\\mathcal\{L\}\_\{dist\}=\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)\+\\alpha\\sum\\limits\_\{\\mathbf\{g\}\_\{t\}\\in V\_\{d\}\}\\mathcal\{L\}\_\{\\text\{KL\}\}\(\\mathbf\{q\}\_\{t\}\\parallel\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)\.\(3\)
A Closer Look at Numerical Learning\. It can be deduced that the above two losses are conceptually equivalent to the following form of loss function:

ℒN=ℋ​\(𝐩,𝐠\)\+α​∑𝐠t∈VdC\(𝐩^\(⋅\|x<t\)\)⏟criterion term⋅D​\(Vd,𝐠t\)⏟distance term,\\mathcal\{L\}\_\{N\}=\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)\+\\alpha\\sum\\limits\_\{\\mathbf\{g\}\_\{t\}\\in V\_\{d\}\}\\underbrace\{C\(\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)\}\_\{\\text\{criterion term\}\}\\cdot\\underbrace\{D\(V\_\{d\},\\mathbf\{g\}\_\{t\}\)\}\_\{\\text\{distance term\}\},\(4\)whereC\(𝐩^\(⋅\|x<t\)\)C\(\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)optimizes the criterion between the predicted digit probability𝐩^\(⋅\|x<t\)\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)and the ground\-truth digit𝐠t\\mathbf\{g\}\_\{t\};D​\(Vd,𝐠t\)D\(V\_\{d\},\\mathbf\{g\}\_\{t\}\)measures the distance between the digit vocabularyVdV\_\{d\}and𝐠t\\mathbf\{g\}\_\{t\}\. Specifically, NTL \(Eq\.[1](https://arxiv.org/html/2605.20369#S3.E1)\) attempts to minimize the probabilities for the non\-target tokens by definingC\(𝐩^\(⋅\|x<t\)\)=𝐩^\(⋅\|x<t\)C\(\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)=\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\. Then, the distance termD​\(Vd,𝐠t\)D\(V\_\{d\},\\mathbf\{g\}\_\{t\}\)guides the model to capture how far each numeric token is from its ground\-truth value\. In DIST2Loss, expanding the penalty termℒKL\(𝐪t∥𝐩^\(⋅\|x<t\)\)\\mathcal\{L\}\_\{\\text\{KL\}\}\(\\mathbf\{q\}\_\{t\}\\parallel\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)and ignoring the zero gradient term, one can get:

ℒKL\(𝐪t∥𝐩^\(⋅\|x<t\)\)=ℋ\(𝐪t,𝐩^\(⋅\|x<t\)\)=−𝐪t⋅log𝐩^\(⋅\|x<t\),\\vskip 2\.0pt\\mathcal\{L\}\_\{\\text\{KL\}\}\(\\mathbf\{q\}\_\{t\}\\parallel\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)=\\mathcal\{H\}\(\\mathbf\{q\}\_\{t\},\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)=\-\\mathbf\{q\}\_\{t\}\\cdot\\log\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\),\\\\\(5\)which implies thatC\(𝐩^\(⋅\|x<t\)\)=−log𝐩^\(⋅\|x<t\)C\(\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)=\-\\log\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)andD​\(Vd,𝐠t\)=𝐪tD\(V\_\{d\},\\mathbf\{g\}\_\{t\}\)=\\mathbf\{q\}\_\{t\}\.

It can be seen that both losses penalize the prediction errors for the non\-target digit tokens\. Comparing the two losses, we see that the key difference lies in the distance term\. In NTL, the distance term penalizes the criterion term by thel1l\_\{1\}distance between the digit vocabulary and the ground\-truth digit, while the distance term in DIST2Loss weights the distance with a soft distance\-based distribution\. More analysis on numerical learning can be found inAppendix[A](https://arxiv.org/html/2605.20369#A1)\.

Conclusion\. In spite of the differences between NTL and DIST2Loss, it can be concluded that they share the same optimization paradigm, which we callcriterion\-distance numerical learning, as laid down in Eq\.[4](https://arxiv.org/html/2605.20369#S3.E4)\. We use the term "numerical learning" here instead of "numerical reasoning" because it is task\-specific, and the positive effects of optimization is hard to generalize to capabilities outside the task domain\. For example, when the model is about to generate a numerical answer, e\.g\., predicting "7", numerical learning can help the model to predict it, and "6" and "8" can be considered very close\. But it cannot help the language model for mathematical reasoning \(e\.g\., the addition and the multiplication between two arbitrary numbers\) and the understanding of structured expression and unseen numbersπ\\pi,∞\\infty, etc\. Essentially, numerical reasoning requires language models to possess a range of capabilities, such as logical reasoning, symbolic comprehension, unit conversion, problem decomposition, contextual semantic association, among which numerical generation accuracy is one of them\. This is what numerical learning aims to explore in this paper\.

## 4 Digit Entropy Loss

In the last section, we have shown that state\-of\-the\-art numerical learning methods NTL and DIST2Loss follow the criterion\-distance optimization paradigm\. Their core design relies on manually defined numerical distance terms to inject an inductive bias of numerical proximity into language models\. This on one hand leads to over\-sharpened or over\-flattened digit distributions\. On the other hand, the manually defined numerical distance may not fully align with the semantic information of digits learned by LLMs\. To address these issues, we introduceDigitEntropyLoss \(DEL\) for better numerical learning\.

From Criterion\-Distance to Digit Entropy Optimization\. Eq\.[4](https://arxiv.org/html/2605.20369#S3.E4)shows that the criterion term is the key to the entire formulation since it provides the gradient flow for the language model to learn\. We draw inspiration from\[[60](https://arxiv.org/html/2605.20369#bib.bib56)\], which reveals that the high entropy tokens acts as pivotal decision points in CoTs, whereas the low entropy tokens indicate determinacy\. We argue that highly deterministic number generation is crucial for language models as it is an indispensable prerequisite for reliable LLM deployment in downstream tasks\. An intuitive way to minimize the digit entropy is to take the digit conditional entropy as the new criterion term for numerical learning, which can be written as:

ℒD​E=ℋ\(𝐩,𝐠\)\+α∑𝐠t∈Vdℋ\(𝐩^\(⋅\|x<t\)\)=ℋ\(𝐩,𝐠\)−α∑𝐠t∈Vd𝐩^\(⋅\|x<t\)log𝐩^\(⋅\|x<t\),\\mathcal\{L\}\_\{DE\}=\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)\+\\alpha\\sum\\limits\_\{\\mathbf\{g\}\_\{t\}\\in V\_\{d\}\}\\mathcal\{H\}\(\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)=\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)\-\\alpha\\sum\\limits\_\{\\mathbf\{g\}\_\{t\}\\in V\_\{d\}\}\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\\log\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\),\(6\)whereℋ​\(𝐩,𝐠\)\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)is the standard cross\-entropy loss, andℋ\(𝐩^\(⋅\|x<t\)\)\\mathcal\{H\}\(\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\)is the Shannon entropy\[[46](https://arxiv.org/html/2605.20369#bib.bib69)\]of the conditional probability distribution of digit tokens\. However, minimizing the digit conditional entropy is unsupervised, which may yield many trivial solutions\. Consequently, this objective encourages the model to produce sharper probability distributions, but it cannot ensure that the probability mass is optimized to the target digits\.

Digit Entropy Loss\. To avoid the confusion of infinite solutions in digit entropy optimization, we propose to jointly optimize the product of the digit conditional probability and the binary cross\-entropy, which is given by:

ℒD​E​L\\displaystyle\\mathcal\{L\}\_\{DEL\}=ℋ\(𝐩,𝐠\)\+α∑𝐠t∈Vd𝐩^\(⋅\|x<t\)𝐁𝐂𝐄\(𝐩˙\(⋅\|x<t\),yt\)\\displaystyle=\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)\+\\alpha\\sum\\limits\_\{\\mathbf\{g\}\_\{t\}\\in V\_\{d\}\}\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\\mathbf\{BCE\}\(\\dot\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\),y\_\{t\}\)\(7\)=ℋ\(𝐩,𝐠\)−α∑𝐠t∈Vd𝐩^\(⋅\|x<t\)\(ytlog𝐩˙\(⋅\|x<t\)\+\(1−yt\)log\(1−𝐩˙\(⋅\|x<t\)\)\),\\displaystyle=\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)\-\\alpha\\sum\\limits\_\{\\mathbf\{g\}\_\{t\}\\in V\_\{d\}\}\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\\Big\(y\_\{t\}\\log\\dot\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\+\(1\-y\_\{t\}\)\\log\\big\(1\-\\dot\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\\big\)\\Big\),whereyt∈ℝ10y\_\{t\}\\in\\mathbb\{R\}^\{10\}is the one\-hot label vector of𝐠t\\mathbf\{g\}\_\{t\}, and𝐩˙\(⋅\|x<t\)=σ\(𝐌Θ\(⋅\|x<t\)\|Vd\)\\dot\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)=\\sigma\(\\mathbf\{M\}\_\{\\Theta\}\(\\cdot\|x\_\{<t\}\)\|\_\{V\_\{d\}\}\)is the digit conditional probability for binary classification, calculated by the sigmoid functionσ\\sigma\. In the formulation, the penalty term of DEL has a global minima at𝐩^​\(𝐠t\|x<t\)=1\\hat\{\\mathbf\{p\}\}\(\\mathbf\{g\}\_\{t\}\|x\_\{<t\}\)=1, and0for anyxt≠𝐠tx\_\{t\}\\neq\\mathbf\{g\}\_\{t\}\. Compared to NTL, DEL sharpens the learned digit distribution in a moderate manner\. Specifically, the product𝐩^\(⋅\|x<t\)⋅𝐁𝐂𝐄\(⋅\)\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\\cdot\\mathbf\{BCE\}\(\\cdot\)imposes a soft constraint: the BCE term’s gradient weakens as non\-target probabilities decrease, i\.e\.,log⁡\(1−𝐩˙​\(v\|x<t\)\)→0\\log\(1\-\\dot\{\\mathbf\{p\}\}\(v\|x\_\{<t\}\)\)\\to 0, while the penalty term of NTL is a constant based on numerical distance, which sustainably pushes the non\-target probabilities to0\.

Deprecation of the Distance Term\. The distance term in previous works\[[67](https://arxiv.org/html/2605.20369#bib.bib9),[10](https://arxiv.org/html/2605.20369#bib.bib36)\]acts for two purposes: weighting the gradients and mimicking the proximity between digits\. It imposes a geometric prior, which specifies that the language model must conform to numerical distance\. However, digits are not merely numerical values on a number line\. For many natural language scenarios, digits serves as symbols, e\.g\., dates, addresses, serial identities, and convey degree, etc\. It has been found that digits learned by LLMs exhibit an entangled representation, blended by both string\-like and numerical representations\[[34](https://arxiv.org/html/2605.20369#bib.bib71)\]\. Shao et al\.\[[47](https://arxiv.org/html/2605.20369#bib.bib72)\]observed Benford’s Curse in the first place of numbers caused by digit bias in web\-collected corpora\. To investigate the semantic structure of digits, we analyze the digital embeddings learned by LLMs\.

For each digitxi∈Vdx\_\{i\}\\in V\_\{d\}, we feed it to an LLM and obtain an embeddingeie\_\{i\}from the last hidden state of transformer\. Fig\.[2](https://arxiv.org/html/2605.20369#S4.F2)Leftshows the t\-SNE visualization\[[56](https://arxiv.org/html/2605.20369#bib.bib70)\]of digit token embeddings on 2D projection surface\. It can be seen that the digit distribution is scattered with weak numerical order\. For instance, the digit 0 is not the closest to 1, but close to 8\. Fig\.[2](https://arxiv.org/html/2605.20369#S4.F2)MiddleandRightshow the differences between the digit token embedding similarity and the ideal numerical distance \(inl1l\_\{1\}\-norm\)\. It is worth noting that these phenomena are hold for LLMs, though the distribution of digits might differ\. This indicates that numerical distance may not be fully aligned with the semantic information learned by LLMs\. In other words, for example, if an LLM treats the digits 2 and 3 as strings, the summation of the two digits may be 23 rather than 5\. This suggests that the manually designed numerical distance may be sub\-optimal and provide contradictive optimization for the model\. Thus, we propose to deprecate the distance term\. Our empirical results later provide the primary support for our design\.

Tab\.[1](https://arxiv.org/html/2605.20369#S4.T1)summarizes the optimization objectives of NTL, DIST2Loss and our proposed DEL\. We can see that both NTL and DIST2Loss rely on the distance term to shape the digit distribution, while our DEL derives from entropy optimization, which is free from the manually defined distance term\. The introduced binary digit probabilities also offer a new optimization path for numerical learning\.

Table 1:The optimization objectives of NTL, DIST2Loss and our DEL\.\\begin\{overpic\}\[width=433\.62pt\]\{pic/digital\-correlation\} \\put\(9\.0,27\.8\)\{tSNE of Digital Embeddings\} \\put\(37\.0,27\.8\)\{Cosine Distance of Digital Embeddings\} \\put\(75\.0,27\.8\)\{Ideal Numerical Distance\} \\end\{overpic\}Figure 2:Visualization of digit token embeddings and their similarity structure on Qwen\-2\.5\-7B\[[39](https://arxiv.org/html/2605.20369#bib.bib29)\]\.Left: t\-SNE 2D projection\[[56](https://arxiv.org/html/2605.20369#bib.bib70)\]of digit token embeddings\.Middle: Cosine distance heatmap of digit embeddings\.Right: Ideal numerical distance heatmap\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]for comparison\.Floating\-point Number Optimization\. In real\-world scenarios, numbers are often dominated by multi\-digits\. The most common examples are multi\-digit integers and floating\-point numbers\. Given a numerical token series:

𝐠=𝐠0L​𝐠1L−1​⋯​𝐠L0​\.​𝐠L\+2−1​𝐠L\+3−2​⋯​𝐠L\+M\+1−M,\\mathbf\{g\}=\\mathbf\{g\}\_\{0\}^\{L\}\\mathbf\{g\}\_\{1\}^\{L\-1\}\\cdots\\mathbf\{g\}\_\{L\}^\{0\}\\textbf\{\.\}\\mathbf\{g\}\_\{L\+2\}^\{\-1\}\\mathbf\{g\}\_\{L\+3\}^\{\-2\}\\cdots\\mathbf\{g\}\_\{L\+M\+1\}^\{\-M\},\(8\)which represents a multi\-digit floating point number𝐠\\mathbf\{g\}, with the highest integer place𝐠0L\\mathbf\{g\}\_\{0\}^\{L\}, the unit place𝐠L0\\mathbf\{g\}\_\{L\}^\{0\}, the tenths place𝐠L\+2−1\\mathbf\{g\}\_\{L\+2\}^\{\-1\}, and the lowest decimal place𝐠L\+M\+2−M\\mathbf\{g\}\_\{L\+M\+2\}^\{\-M\}\. It has been discussed in\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]that extending DIST2Loss to multi\-digit integers may face the problem of credit assignment\. This is because the global distance metric is the result of the collective action of all tokens and cannot be simply broken down into the independent distance of each token\. To alleviate this issue, DIST2Loss integrates the place value weighting to allocate more penalty to higher place digits\. However, this strategy fails when dealing with floating\-point numbers, because the decimal will be treated as a new integer, resulting in discontinuous penalty\. We therefore propose floating\-point number optimization, which delivers larger penalty for integers, and a small penalty for decimals\. We suggest a linear penalty for integer places and an exponential decay for decimals:

u\(t\)=\{k​i\+1,for​𝐠ti,i≥0,βi,for​𝐠ti,i<0,u\(t\)=\\left\\\{\\begin\{aligned\} &ki\+1,&\\text\{for \}\\mathbf\{g\}\_\{t\}^\{i\},\\,i\\geq 0,\\\\ &\\beta^\{i\},&\\text\{for \}\\mathbf\{g\}\_\{t\}^\{i\},\\,i<0,\\end\{aligned\}\\right\.\(9\)where parametersk,β\>0k,\\beta\>0control the penalty amplitude for each token\. The visual comparison ofu​\(t\)u\(t\)in DIST2Loss and DEL can be seen in Fig\.[C\.1](https://arxiv.org/html/2605.20369#A3.F1)ofAppendix[B](https://arxiv.org/html/2605.20369#A2)\. Finally, we generalize DEL from single\-digit optimization to floating\-point number optimization by rewriting Eq\.[7](https://arxiv.org/html/2605.20369#S4.E7)as follows:

ℒD​E​L=ℋ\(𝐩,𝐠\)\+α∑𝐠t∈Vdu\(t\)𝐩^\(⋅\|x<t\)𝐁𝐂𝐄\(𝐩˙\(⋅\|x<t\),yt\)\.\\mathcal\{L\}\_\{DEL\}=\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)\+\\alpha\\sum\\limits\_\{\\mathbf\{g\}\_\{t\}\\in V\_\{d\}\}u\(t\)\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)\\mathbf\{BCE\}\(\\dot\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\),y\_\{t\}\)\.\(10\)

## 5 Experiments

In this section, we conduct extensive experiments to demonstrate the performance of the proposed DEL\. We apply DEL to several recently popular LLMs, and evaluate it together with state\-of\-the\-art losses on the classic mathematical reasoning and code generation tasks\.

### 5\.1Experimental Setup

Baseline and Compared Methods\. We conduct experiments using four widely used open\-source LLMs, including CodeLlama\[[44](https://arxiv.org/html/2605.20369#bib.bib15)\], Mistral\[[23](https://arxiv.org/html/2605.20369#bib.bib23)\], Qwen2\.5\[[39](https://arxiv.org/html/2605.20369#bib.bib29)\], and DeepSeek\-Math\-instruct\[[17](https://arxiv.org/html/2605.20369#bib.bib19)\]\. MLE is used as the baseline loss\. To make the comparison clean and fair, all experimental settings irrelevant to the loss function remain unchanged for a given LLM\. We compare DEL with the following representative methods: MixCE\[[68](https://arxiv.org/html/2605.20369#bib.bib38)\]and EMO\[[43](https://arxiv.org/html/2605.20369#bib.bib39)\], which aim at improving language understanding on general texts, and the recently developed numerical learning methods NTL\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]and DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]\. In addition, we also evaluate 12 off\-the\-shelf open\-source LLMs, which have made remarkable progress in mathematical reasoning\.

Table 2:Performance comparison of mainstream LLMs trained using their own losses, the general approaches MixCE, EMO, and numerical learning methods NTL, DIST222^\{2\}Loss and our DEL\. Best results are highlighted inbold\.MethodBasemACCBenchmarkReferenceGSM8KMATHSVAMPSimulEqAQuASATMMLUGalactica\-6\.7B\[[50](https://arxiv.org/html/2605.20369#bib.bib10)\]GAL16\.210\.22\.225\.64\.225\.617\.528\.0Arxiv22Vicuna\-1\.5\-7B\[[69](https://arxiv.org/html/2605.20369#bib.bib11)\]Llama\-220\.213\.14\.936\.64\.927\.225\.930\.4NeurIPS23Llama\-2\-7B\[[55](https://arxiv.org/html/2605.20369#bib.bib14)\]\-20\.514\.62\.534\.55\.030\.326\.829\.8Arxiv23CodeLlama\-7B\[[44](https://arxiv.org/html/2605.20369#bib.bib15)\]Llama\-224\.325\.213\.049\.43\.524\.028\.626\.9Arxiv23MetaMath\-7B\[[64](https://arxiv.org/html/2605.20369#bib.bib6)\]Mistral42\.272\.529\.477\.534\.224\.833\.223\.7ICLR24DeepMath\-Zero\-Math\-7B\[[17](https://arxiv.org/html/2605.20369#bib.bib19)\]Qwen2\.5\-Math44\.643\.750\.838\.727\.244\.953\.653\.5Arxiv25Math\-Shepherd\-RL\-7B\[[58](https://arxiv.org/html/2605.20369#bib.bib3)\]Mistral45\.482\.622\.682\.445\.324\.433\.227\.1ACL24Xwin\-Math\-V1\.1\-7B\[[25](https://arxiv.org/html/2605.20369#bib.bib20)\]Llama 245\.574\.732\.381\.050\.626\.427\.426\.0Arxiv24WizardMath\-V1\.1\-7B\[[33](https://arxiv.org/html/2605.20369#bib.bib16)\]Mistral48\.174\.529\.375\.470\.725\.333\.728\.0Arxiv23Skywork\-OR1\-Math\-7B\[[16](https://arxiv.org/html/2605.20369#bib.bib37)\]Qwen2\.5\-Math55\.158\.580\.349\.684\.350\.035\.055\.0Arxiv25MathScale\-7B\[[49](https://arxiv.org/html/2605.20369#bib.bib4)\]Mistral61\.075\.637\.880\.675\.744\.562\.350\.6ICML24DART\-Math\-7B \(Prop2Diff\)\[[52](https://arxiv.org/html/2605.20369#bib.bib5)\]DeepSeek\-math63\.187\.452\.683\.873\.738\.656\.848\.5NeurIPS24PoT code generationCoT reasoningMLECodeLlama\-7B45\.259\.434\.072\.040\.737\.834\.138\.6\-MixCE\[[68](https://arxiv.org/html/2605.20369#bib.bib38)\]CodeLlama\-7B46\.159\.934\.472\.643\.038\.234\.540\.3ACL23EMO\[[43](https://arxiv.org/html/2605.20369#bib.bib39)\]CodeLlama\-7B\-\-\-\-\-\-\-\-ICLR24NTL\-WAS\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]CodeLlama\-7B47\.459\.735\.072\.344\.439\.042\.738\.5ICML25DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]CodeLlama\-7B47\.659\.434\.071\.244\.944\.539\.639\.9ICLR26DEL \(Ours\)CodeLlama\-7B49\.059\.935\.772\.444\.446\.943\.640\.1\-MLEQwen2\.5\-1\.5B52\.854\.042\.371\.644\.245\.356\.455\.5\-MixCE\[[68](https://arxiv.org/html/2605.20369#bib.bib38)\]Qwen2\.5\-1\.5B52\.952\.842\.772\.743\.247\.254\.557\.3ACL23EMO\[[43](https://arxiv.org/html/2605.20369#bib.bib39)\]Qwen2\.5\-1\.5B53\.454\.542\.673\.547\.748\.052\.755\.1ICLR24NTL\-WAS\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]Qwen2\.5\-1\.5B53\.855\.143\.573\.346\.143\.159\.156\.3ICML25DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]Qwen2\.5\-1\.5B53\.154\.241\.973\.244\.645\.755\.956\.5ICLR26DEL \(Ours\)Qwen2\.5\-1\.5B55\.457\.042\.674\.048\.148\.860\.956\.1\-MLEMistral\-7B54\.275\.139\.382\.444\.248\.845\.044\.9\-MixCE\[[68](https://arxiv.org/html/2605.20369#bib.bib38)\]Mistral\-7B55\.175\.639\.481\.945\.949\.447\.146\.6ACL23EMO\[[43](https://arxiv.org/html/2605.20369#bib.bib39)\]Mistral\-7B\-\-\-\-\-\-\-\-ICLR24NTL\-WAS\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]Mistral\-7B55\.675\.139\.983\.544\.949\.249\.546\.8ICML25DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]Mistral\-7B54\.674\.840\.081\.443\.650\.847\.743\.7ICLR26DEL \(Ours\)Mistral\-7B56\.576\.040\.381\.746\.953\.151\.845\.5\-MLEDeepSeek\-math\-7B\-Ins\.64\.476\.851\.480\.952\.759\.169\.160\.9\-MixCE\[[68](https://arxiv.org/html/2605.20369#bib.bib38)\]DeepSeek\-math\-7B\-Ins\.64\.677\.251\.281\.254\.955\.970\.161\.9ACL23EMO\[[43](https://arxiv.org/html/2605.20369#bib.bib39)\]DeepSeek\-math\-7B\-Ins\.64\.877\.650\.980\.754\.757\.969\.562\.1ICLR24NTL\-WAS\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]DeepSeek\-math\-7B\-Ins\.64\.977\.051\.781\.650\.259\.473\.660\.8ICML25DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]DeepSeek\-math\-7B\-Ins\.64\.876\.651\.282\.553\.758\.369\.561\.7ICLR26DEL \(Ours\)DeepSeek\-math\-7B\-Ins\.66\.177\.651\.983\.052\.561\.074\.562\.1\-MLEQwen2\.5\-7B68\.381\.451\.687\.051\.063\.876\.866\.3\-MixCE\[[68](https://arxiv.org/html/2605.20369#bib.bib38)\]Qwen2\.5\-7B68\.381\.655\.386\.953\.561\.472\.766\.5ACL23EMO\[[43](https://arxiv.org/html/2605.20369#bib.bib39)\]Qwen2\.5\-7B68\.581\.455\.887\.650\.263\.075\.066\.2ICLR24NTL\-WAS\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]Qwen2\.5\-7B68\.881\.554\.687\.153\.559\.879\.165\.9ICML25DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\]Qwen2\.5\-7B69\.281\.756\.087\.051\.863\.876\.867\.6ICLR26DEL \(Ours\)Qwen2\.5\-7B70\.681\.756\.087\.657\.065\.478\.667\.9\-Datasets\. We evaluate DEL on seven widely used mathematical reasoning benchmarks: 1\) high quality grade school math problems dataset GSM8K\[[11](https://arxiv.org/html/2605.20369#bib.bib77)\]; 2\) multi\-level difficulty math competition dataset MATH\[[19](https://arxiv.org/html/2605.20369#bib.bib78)\]; 3\) arithmetic math word problem dataset SVAMP\[[38](https://arxiv.org/html/2605.20369#bib.bib79)\]; 4\) simultaneous equations dataset SimulEq\[[24](https://arxiv.org/html/2605.20369#bib.bib80)\]; 5\) algebraic word problems with rationales AQuA\[[30](https://arxiv.org/html/2605.20369#bib.bib74)\]; 6\) general college entrance exams SAT\-Math\[[70](https://arxiv.org/html/2605.20369#bib.bib75)\]; and 7\) college math problems dataset MMLU\[[18](https://arxiv.org/html/2605.20369#bib.bib76)\]\. We report zero\-shot in\-context\-learning task accuracy and the mean accuracy \(mACC\) over the evaluated benchmarks\. To evaluate the applicability, we adopt CoT reasoning for AQuA, SAT\-Math, and MMLU, while we use program\-of\-thought \(PoT\) for code generation on GSM8K, MATH, SVAMP, and SimulEq\. If the code generation fails, we follow\[[66](https://arxiv.org/html/2605.20369#bib.bib1)\]to use CoT reasoning for assistant\. Training is conducted on the large\-scale high\-quality math instruction tuning dataset MathInstruct\[[66](https://arxiv.org/html/2605.20369#bib.bib1)\], which consists of 13 math datasets covering a broad variety of math fields and complexity levels such as Algebra, Probability, NumTheory, Calculus, and Geometry\.

Implementation Details\. We adopt the Huggingface transformer library\[[62](https://arxiv.org/html/2605.20369#bib.bib73)\]to conduct the experiments\. We train all the models for 2 epochs\. The AdamW\[[32](https://arxiv.org/html/2605.20369#bib.bib81)\]is adopted as the optimizer with bfloat16 precision enabled\. We use a learning rate of 2e\-5 for CodeLlama, and 5e\-6 for Mistral, Qwen2\.5, and DeepSeek\-Math\-Instruct\. The learning rate decay adopts a cosine scheduler with a warm\-up strategy for the first 3% training steps\. The hyperparameterα\\alphain Eq\.[10](https://arxiv.org/html/2605.20369#S4.E10)is tuned separately for LLMs due to their different vocabulary size, which is set to0\.10\.1for CodeLlama and Mistral, and0\.010\.01for Qwen2\.5 and DeepSeek\-Math\-Instruct\. We setk=2k=2andβ=1\.02\\beta=1\.02in the floating\-point number optimization by default\. The ablation for these parameters can be found inAppendix[B](https://arxiv.org/html/2605.20369#A2)\. All the trainings are conducted on 8 NVIDIA A100 GPUs with batchsize 1 per GPU and gradient accumulation step 32, thus resulting in a total batch size of 256\.

### 5\.2Experimental Results

This subsection shows the results of DEL on four PoT code generation settings \(GSM8K, MATH, SVAMP, and SimulEq\) and three CoT reasoning settings \(AQuA, SAT, and MMLU\)\. For other evaluations, we use Qwen2\.5\-1\.5B by default\. InAppendix[C](https://arxiv.org/html/2605.20369#A3), we provide more exempler visual comparisons for various numerical learning methods\.

Table 3:Numerical answer evaluation on GSM8K with various loss functions\. We report the numerical distance and the task accuracy for the predicted number answers, along with the per digit metric for the unit place, tens place, hundreds place, thousands place, and ten thousands place, denoted by11,10110^\{1\},10210^\{2\},10310^\{3\},10410^\{4\}, respectively\.\(a\)Numerical distance evaluation\.
\(b\)Numerical accuracy evaluation\.

Table 4:Evaluation of diverged components in numerical learning\.Benchmark Evaluation\. We report the quantitative results in Tab\.[2](https://arxiv.org/html/2605.20369#S5.T2)\. One can see that DEL produces the best results in most benchmarks and achieves the best average performance across all model architectures, demonstrating consistent improvements over existing methods\. It can be seen that the general methods MixCE and EMO can improve mathematical reasoning performance by enhancing logical deduction and language understanding capability\. While EMO achieves better overall results than MixCE, it fails in training on CodeLlama and Mistral\. This is presumably because that EMO relies on the token embeddings to calculate the similarity between tokens, which may introduce numerical instability in training\. The numerical learning methods NTL and DIST2Loss better boost the accuracy since they are designed to improve digit prediction performance\. Our DEL better caters to the low\-entropy nature of digits, resulting in the best numerical learning performance\.

Numerical answers evaluation\. Aside from global task accuracy, we also compare the numerical answers for the methods\. We consider two evaluation metric\. In Tab\.[3a](https://arxiv.org/html/2605.20369#S5.T3.sf1), we calculate the global numerical distance for the predicted numbers, as well as for every digital place\. To alleviate the effect of large numbers, we follow\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]to use the log Mean Absolute Error \(MAE\) to measure the relative distance between the predicted digits and the ground\-truths\. We omit reporting very large digital place because the model may predict excessively long digital sequences, and these numbers are rare\. In Tab\.[3b](https://arxiv.org/html/2605.20369#S5.T3.sf2), we report the numerical accuracy for the predicted numbers, as well as for every digital place\. The results shows that our DEL can substantially reduce the numerical distance even without the explicit injection of numerical distance term, demonstrating a more precise number generation\.

Evaluation of diverse components in numerical learning\. We conduct experiments to check out the effect of each component in numerical learning\. As shown in Tab\.[4](https://arxiv.org/html/2605.20369#S5.T4), we start from MLE, a clean baseline of 52\.9 mACC on seven benchmarks\. The second row is NTL\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\], which adopts the digital probability𝐩^\(⋅\|x<t\)\\hat\{\\mathbf\{p\}\}\(\\cdot\|x\_\{<t\}\)as the criterion term, along with the numerical distance, achieving a mACC of 53\.8\. It can be further improved to 54\.4 if we intergrate our floating\-point number generation into NTL, as shown by the third row\. Then, in 4\-th row, our DEL without any other components produces 55\.1 mACC, demonstrating the superiority of the new criterion term\. If we add the distance term of NTL, it degrades to 54\.3, as shown in the results of 5\-th row\. As we have analyzed in Sec\.[4](https://arxiv.org/html/2605.20369#S4), the numerical distance terms injects a strong geometric prior into the LLMs, which has a deviation from the semantic information of digits\. Thus we deprecate the distance term in our formulation\. Finally, combined with the floating\-point optimization, the performance can be further improved to 55\.4 mACC \(see the last row\)\. This gives us our final DEL formula \(Eq\.[10](https://arxiv.org/html/2605.20369#S4.E10)\)

Number Uncertainty\. To investigate the uncertainty of the predicted numbers, we further statistically analyze the token entropy of the predicted sequence\. Based on the full space probability𝐩\(⋅\|x<t\)\\mathbf\{p\}\(\\cdot\|x\_\{<t\}\), we decompose the entropy calculation into two disjoint parts, the digital part and the non\-digital part\. As shown in Fig\.[3](https://arxiv.org/html/2605.20369#S5.F3), the orginal baseline MLE produces a low digital entropy, approximately 0\.1\. NTL\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\]significantly reduces digital entropy because its optimization goal is to forcibly push the probability of non\-target digits to0\. Another extreme example is DIST2Loss\[[10](https://arxiv.org/html/2605.20369#bib.bib36)\], which significantly increases the digital entropy, making it even exceed11\. This is because DIST2Loss takes the distance\-driven soft targets as optimization objectives\. It should be noted that the average entropy of non\-digit tokens is around0\.3∼0\.50\.3\\sim 0\.5for all these methods, as depicted by the orange line in Fig\.[3](https://arxiv.org/html/2605.20369#S5.F3)\. This indicates that DIST2Loss largely rises the uncertainty of number generation, whereas NTL tends to significantly sharpen the probability distribution of digits\. The criterion term of DEL makes it a mild method for optimizing number uncertainty\.

\\begin\{overpic\}\[width=433\.62pt\]\{pic/digital\-entropy\} \\put\(15\.0,1\.0\)\{\(a\) Digital entropy on SVAMP\} \\put\(62\.0,1\.0\)\{\(b\) Digital entropy on GSM8K\} \\end\{overpic\}Figure 3:Comparison of digital entropy between various numerical learning methods\.

## 6 Conclusion

In this paper, we analyzed the existing numerical learning approaches in large language model training, and found that state\-of\-the\-art methods generally follow the criterion\-distance formulation\. From the perspective of entropy optimization, we presented digit entropy loss \(DEL\) to achieve better performance on number generation\. We further generalized the numerical learning to floating\-point number optimization, enabling the language models with holistic numerical awareness\. Experiments on seven mathematical reasoning benchmarks demonstrated the effectiveness of the proposed DEL over prior state\-of\-the\-art numerical learning methods\.

Limitations\. While demonstrating consistent improvements over previous methods, DEL has some limitations\. Firstly, it is still optimized in a token\-level manner, without joint modeling of the global numerical semantics of the number sequence\. Secondly, DEL currently extends numerical learning to floating\-point number optimization, but has not yet expanded many mathematical operations\. In the future, one may consider modeling mathematical operations in LLMs, such as addition, multiplication, division, exponentiation, etc\., enabling LLMs with more numerical reasoning capabilities\.

## ■\\blacksquareAcknowledgements

This work was supported by the Visual Computing Lab at The Hong Kong Polytechnic University and VCIP at College of Computer Science, Nankai University\. We thank our colleagues for valuable discussions\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15)\.
- \[2\]\(2025\)The unreasonable effectiveness of entropy minimization in LLM reasoning\.InNeurIPS,Cited by:[§A\.2](https://arxiv.org/html/2605.20369#A1.SS2.p4.1)\.
- \[3\]J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15)\.
- \[4\]Y\. Bengio, R\. Ducharme, P\. Vincent, and C\. Jauvin\(2003\)A neural probabilistic language model\.Journal of Machine Learning Research3\(Feb\),pp\. 1137–1155\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[5\]J\. Bian, M\. Feng, W\. Dong, F\. Wu, J\. Luo, Y\. Wang, and G\. Shi\(2025\)Feature information driven position gaussian distribution estimation for tiny object detection\.InCVPR,pp\. 30376–30386\.Cited by:[§A\.2](https://arxiv.org/html/2605.20369#A1.SS2.p4.1)\.
- \[6\]P\. F\. Brown, V\. J\. Della Pietra, P\. V\. Desouza, J\. C\. Lai, and R\. L\. Mercer\(1992\)Class\-based n\-gram models of natural language\.Computational linguistics18\(4\),pp\. 467–480\.Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[7\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.InNeurIPS,pp\. 1877–1901\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1),[§3](https://arxiv.org/html/2605.20369#S3.p1.15)\.
- \[8\]W\. Chen, X\. Ma, X\. Wang, and W\. W\. Cohen\(2023\)Program of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks\.Transactions on Machine Learning Research\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[9\]Z\. Chen, J\. Wu, W\. Wang, W\. Su, G\. Chen, S\. Xing, M\. Zhong, Q\. Zhang, X\. Zhu, L\. Lu,et al\.\(2024\)InternVL: scaling up vision foundation models and aligning for generic visual\-linguistic tasks\.InCVPR,pp\. 24185–24198\.Cited by:[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1)\.
- \[10\]J\. Chung, S\. Kim, Y\. Jo, J\. Park, D\. Min, and Y\. Yu\(2026\)Teaching metric distance to autoregressive multimodal foundational models\.InICLR,Cited by:[Table A\.3](https://arxiv.org/html/2605.20369#A1.T3.1.1.1),[Figure 1](https://arxiv.org/html/2605.20369#S1.F1),[§1](https://arxiv.org/html/2605.20369#S1.p3.3),[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1),[§3](https://arxiv.org/html/2605.20369#S3.p3.2),[§4](https://arxiv.org/html/2605.20369#S4.p4.1),[§4](https://arxiv.org/html/2605.20369#S4.p7.7),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2605.20369#S5.SS2.p5.7),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.3.1.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.4.2.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.5.3.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.6.4.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.5.1),[3a](https://arxiv.org/html/2605.20369#S5.T3.sf1.5.5.1),[3b](https://arxiv.org/html/2605.20369#S5.T3.sf2.5.5.1)\.
- \[11\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p2.1)\.
- \[12\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InNAACL,pp\. 4171–4186\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[13\]L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig\(2023\)PAL: program\-aided language models\.InICML,pp\. 10764–10799\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[14\]A\. Graves, S\. Fernández, F\. Gomez, and J\. Schmidhuber\(2006\)Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks\.InICML,pp\. 369–376\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[15\]A\. Graves and N\. Jaitly\(2014\)Towards end\-to\-end speech recognition with recurrent neural networks\.InICML,pp\. 1764–1772\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[16\]J\. He, J\. Liu, C\. Y\. Liu, R\. Yan, C\. Wang, P\. Cheng, X\. Zhang, F\. Zhang, J\. Xu, W\. Shen,et al\.\(2025\)Skywork open reasoner 1 technical report\.arXiv preprint arXiv:2505\.22312\.Cited by:[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.17.12.1)\.
- \[17\]Z\. He, T\. Liang, J\. Xu, Q\. Liu, X\. Chen, Y\. Wang, L\. Song, D\. Yu, Z\. Liang, W\. Wang, Z\. Zhang, R\. Wang, Z\. Tu, H\. Mi, and D\. Yu\(2025\)DeepMath\-103K: a large\-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning\.arXiv preprint arXiv:2504\.11456\.Cited by:[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p1.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.13.8.1)\.
- \[18\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2021\)Measuring massive multitask language understanding\.InICLR,Cited by:[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p2.1)\.
- \[19\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt\(2021\)Measuring mathematical problem solving with the math dataset\.NeurIPS\.Cited by:[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p2.1)\.
- \[20\]S\. Hochreiter and J\. Schmidhuber\(1997\)Long short\-term memory\.Neural computation9\(8\),pp\. 1735–1780\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[21\]A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. Choi\(2020\)The curious case of neural text degeneration\.InICLR,Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p2.1)\.
- \[22\]H\. Ji, P\. Ke, Z\. Hu, R\. Zhang, and M\. Huang\(2023\)Tailoring language generation models under total variation distance\.InICLR,Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1)\.
- \[23\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed\(2023\)Mistral 7b\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p1.1)\.
- \[24\]R\. Koncel\-Kedziorski, S\. Roy, A\. Amini, N\. Kushman, and H\. Hajishirzi\(2016\)MAWPS: a math word problem repository\.InNAACL,pp\. 1152–1157\.Cited by:[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p2.1)\.
- \[25\]C\. Li, W\. Wang, J\. Hu, Y\. Wei, N\. Zheng, H\. Hu, Z\. Zhang, and H\. Peng\(2024\)Common 7b language models already possess strong math capabilities\.arXiv preprint arXiv:2403\.04706\.Cited by:[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.15.10.1)\.
- \[26\]M\. Li, T\. Lv, J\. Chen, L\. Cui, Y\. Lu, D\. Florencio, C\. Zhang, Z\. Li, and F\. Wei\(2023\)Trocr: transformer\-based optical character recognition with pre\-trained models\.InAAAI,pp\. 13094–13102\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[27\]Q\. Li, H\. Xiao, and L\. Shen\(2025\)BCE vs\. CE in deep feature learning\.InICML,Cited by:[§A\.2](https://arxiv.org/html/2605.20369#A1.SS2.p5.6)\.
- \[28\]Z\. Li, C\. Chen, T\. Xu, Z\. Qin, J\. Xiao, Z\. Luo, and R\. Sun\(2025\)Preserving diversity in supervised fine\-tuning of large language models\.InICLR,Cited by:[§A\.2](https://arxiv.org/html/2605.20369#A1.SS2.p4.1)\.
- \[29\]T\. Lin, P\. Goyal, R\. Girshick, K\. He, and P\. Dollár\(2017\)Focal loss for dense object detection\.InICCV,pp\. 2980–2988\.Cited by:[§A\.2](https://arxiv.org/html/2605.20369#A1.SS2.p5.6)\.
- \[30\]W\. Ling, D\. Yogatama, C\. Dyer, and P\. Blunsom\(2017\)Program induction by rationale generation: learning to solve and explain algebraic word problems\.InACL,pp\. 158–167\.Cited by:[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p2.1)\.
- \[31\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.InNeurIPS,pp\. 34892–34916\.Cited by:[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1)\.
- \[32\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InICLR,Cited by:[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p3.5)\.
- \[33\]H\. Luo, Q\. Sun, C\. Xu, P\. Zhao, J\. Lou, C\. Tao, X\. Geng, Q\. Lin, S\. Chen, and D\. Zhang\(2023\)Wizardmath: empowering mathematical reasoning for large language models via reinforced evol\-instruct\.arXiv preprint arXiv:2308\.09583\.Cited by:[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.16.11.1)\.
- \[34\]R\. Marjieh, V\. Veselovsky, T\. L\. Griffiths, and I\. Sucholutsky\(2025\)What is a number, that a large language model may know it?\.InNeurIPS Workshop,Cited by:[§4](https://arxiv.org/html/2605.20369#S4.p4.1)\.
- \[35\]T\. Mikolov, K\. Chen, G\. Corrado, and J\. Dean\(2013\)Efficient estimation of word representations in vector space\.arXiv preprint arXiv:1301\.3781\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[36\]C\. Mitra, B\. Huang, T\. Darrell, and R\. Herzig\(2024\)Compositional chain\-of\-thought prompting for large multimodal models\.InCVPR,pp\. 14420–14431\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[37\]F\. J\. Och\(2003\)Minimum error rate training in statistical machine translation\.InACL,pp\. 160–167\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[38\]A\. Patel, S\. Bhattamishra, and N\. Goyal\(2021\)Are nlp models really able to solve simple math word problems?\.InNAACL,pp\. 2080–2094\.Cited by:[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p2.1)\.
- \[39\]Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu\(2025\)Qwen2\.5 technical report\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15),[Figure 2](https://arxiv.org/html/2605.20369#S4.F2),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p1.1)\.
- \[40\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever\(2023\)Robust speech recognition via large\-scale weak supervision\.InICML,pp\. 28492–28518\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[41\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15)\.
- \[42\]C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu\(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of Machine Learning Research21\(140\),pp\. 1–67\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15)\.
- \[43\]S\. Ren, Z\. Wu, and K\. Q\. Zhu\(2024\)EMO: earth mover distance optimization for auto\-regressive language modeling\.InICLR,Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p1.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.23.18.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.28.23.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.33.28.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.38.33.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.43.38.1),[3a](https://arxiv.org/html/2605.20369#S5.T3.sf1.5.9.3.1),[3b](https://arxiv.org/html/2605.20369#S5.T3.sf2.5.9.3.1)\.
- \[44\]B\. Roziere, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan, Y\. Adi, J\. Liu, R\. Sauvestre, T\. Remez,et al\.\(2023\)Code Llama: open foundation models for code\.arXiv preprint arXiv:2308\.12950\.Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1),[§3](https://arxiv.org/html/2605.20369#S3.p1.15),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p1.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.11.6.1)\.
- \[45\]R\. Sennrich, B\. Haddow, and A\. Birch\(2016\)Neural machine translation of rare words with subword units\.InACL,pp\. 1715–1725\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[46\]C\. E\. Shannon\(1948\)A mathematical theory of communication\.The Bell System Technical Journal27\(3\),pp\. 379–423\.Cited by:[§4](https://arxiv.org/html/2605.20369#S4.p2.2)\.
- \[47\]J\. Shao, Y\. Lu, and J\. Yang\(2025\)Benford’s curse: tracing digit bias to numerical hallucination in LLMs\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§4](https://arxiv.org/html/2605.20369#S4.p4.1)\.
- \[48\]F\. Shi, X\. Chen, K\. Misra, N\. Scales, D\. Dohan, E\. H\. Chi, N\. Schärli, and D\. Zhou\(2023\)Large language models can be easily distracted by irrelevant context\.InICML,pp\. 31210–31227\.Cited by:[Appendix C](https://arxiv.org/html/2605.20369#A3.p2.1)\.
- \[49\]Z\. Tang, X\. Zhang, B\. Wang, and F\. Wei\(2024\)MathScale: scaling instruction tuning for mathematical reasoning\.InICML,pp\. 47885–47900\.Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p1.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.18.13.1)\.
- \[50\]R\. Taylor, M\. Kardas, G\. Cucurull, T\. Scialom, A\. Hartshorn, E\. Saravia, A\. Poulton, V\. Kerkez, and R\. Stojnic\(2022\)Galactica: a large language model for science\.arXiv preprint arXiv:2211\.09085\.Cited by:[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.8.3.1)\.
- \[51\]Q\. Teamet al\.\(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15)\.
- \[52\]Y\. Tong, X\. Zhang, R\. Wang, R\. Wu, and J\. He\(2024\)DART\-Math: difficulty\-aware rejection tuning for mathematical problem\-solving\.InNeurIPS,Vol\.37,pp\. 7821–7846\.Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p1.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.19.14.1)\.
- \[53\]S\. Toshniwal, I\. Moshkov, S\. Narenthiran, D\. Gitman, F\. Jia, and I\. Gitman\(2024\)Openmathinstruct\-1: a 1\.8 million math instruction tuning dataset\.InNeurIPS,Vol\.37,pp\. 34737–34774\.Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p1.1)\.
- \[54\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15)\.
- \[55\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.10.5.1)\.
- \[56\]L\. Van der Maaten and G\. Hinton\(2008\)Visualizing data using t\-sne\.Journal of Machine Learning Research9\(11\)\.Cited by:[Figure 2](https://arxiv.org/html/2605.20369#S4.F2),[§4](https://arxiv.org/html/2605.20369#S4.p5.3)\.
- \[57\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.NeurIPS\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[58\]P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui\(2024\)Math\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InACL,pp\. 9426–9439\.Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p1.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.14.9.1)\.
- \[59\]P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.\(2024\)Qwen2\-VL: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1)\.
- \[60\]S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang,et al\.\(2025\)Beyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for llm reasoning\.InNeurIPS,Cited by:[§A\.2](https://arxiv.org/html/2605.20369#A1.SS2.p4.1),[§1](https://arxiv.org/html/2605.20369#S1.p4.1),[§4](https://arxiv.org/html/2605.20369#S4.p2.3)\.
- \[61\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InNeurIPS,pp\. 24824–24837\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[62\]T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz,et al\.\(2020\)Transformers: state\-of\-the\-art natural language processing\.InEMNLP,pp\. 38–45\.Cited by:[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p3.5)\.
- \[63\]S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. Narasimhan\(2023\)Tree of thoughts: deliberate problem solving with large language models\.InNeurIPS,pp\. 11809–11822\.Cited by:[§2\.1](https://arxiv.org/html/2605.20369#S2.SS1.p1.1)\.
- \[64\]L\. Yu, W\. Jiang, H\. Shi, J\. YU, Z\. Liu, Y\. Zhang, J\. Kwok, Z\. Li, A\. Weller, and W\. Liu\(2024\)MetaMath: bootstrap your own mathematical questions for large language models\.InICLR,Cited by:[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.12.7.1)\.
- \[65\]Z\. Yuan, H\. Yuan, C\. Tan, W\. Wang, and S\. Huang\(2023\)How well do large language models perform in arithmetic tasks?\.arXiv preprint arXiv:2304\.02015\.Cited by:[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1)\.
- \[66\]X\. Yue, X\. Qu, G\. Zhang, Y\. Fu, W\. Huang, H\. Sun, Y\. Su, and W\. Chen\(2024\)Mammoth: building math generalist models through hybrid instruction tuning\.InICLR,Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p2.1)\.
- \[67\]J\. Zausinger, L\. Pennig, A\. Kozina, S\. Sdahl, J\. Sikora, A\. Dendorfer, T\. Kuznetsov, M\. Hagog, N\. Wiedemann, K\. Chlodny,et al\.\(2025\)Regress, don’t guess–a regression\-like loss on number tokens for language models\.InICML,Cited by:[§A\.2](https://arxiv.org/html/2605.20369#A1.SS2.p1.3),[Table A\.3](https://arxiv.org/html/2605.20369#A1.T3.1.5.3.1),[Figure 1](https://arxiv.org/html/2605.20369#S1.F1),[§1](https://arxiv.org/html/2605.20369#S1.p3.3),[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1),[§3](https://arxiv.org/html/2605.20369#S3.p2.6),[§3](https://arxiv.org/html/2605.20369#S3.p2.7),[Figure 2](https://arxiv.org/html/2605.20369#S4.F2),[§4](https://arxiv.org/html/2605.20369#S4.p4.1),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2605.20369#S5.SS2.p3.1),[§5\.2](https://arxiv.org/html/2605.20369#S5.SS2.p4.1),[§5\.2](https://arxiv.org/html/2605.20369#S5.SS2.p5.7),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.24.19.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.29.24.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.34.29.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.39.34.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.44.39.1),[3a](https://arxiv.org/html/2605.20369#S5.T3.sf1.5.10.4.1),[3b](https://arxiv.org/html/2605.20369#S5.T3.sf2.5.10.4.1)\.
- \[68\]S\. Zhang, S\. Wu, O\. Irsoy, S\. Lu, M\. Bansal, M\. Dredze, and D\. Rosenberg\(2023\)MixCE: training autoregressive language models by mixing forward and reverse cross\-entropies\.InACL,pp\. 9027–9050\.Cited by:[§1](https://arxiv.org/html/2605.20369#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.20369#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p1.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.22.17.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.27.22.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.32.27.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.37.32.1),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.42.37.1),[3a](https://arxiv.org/html/2605.20369#S5.T3.sf1.5.8.2.1),[3b](https://arxiv.org/html/2605.20369#S5.T3.sf2.5.8.2.1)\.
- \[69\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InNeurIPS,Vol\.36,pp\. 46595–46623\.Cited by:[§3](https://arxiv.org/html/2605.20369#S3.p1.15),[Table 2](https://arxiv.org/html/2605.20369#S5.T2.7.9.4.1)\.
- \[70\]W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. Duan\(2024\)AGIEval: a human\-centric benchmark for evaluating foundation models\.InNAACL,pp\. 2299–2314\.Cited by:[§5\.1](https://arxiv.org/html/2605.20369#S5.SS1.p2.1)\.

Appendix

In appendix, we provide the following materials:

1. A\.More analysis for various numerical learning methods \(referring to Sec\.[3](https://arxiv.org/html/2605.20369#S3)in the main paper\)\.
2. B\.Hyper\-parameters study for the loss weightα\\alphaand the designs in floating\-point number optimization \(referring to Sec\.[5\.1](https://arxiv.org/html/2605.20369#S5.SS1)in the main paper\)\.
3. C\.Exempler visual comparisons for the numerical learning methods, along with our failure case analysis \(referring to Sec\.[5\.2](https://arxiv.org/html/2605.20369#S5.SS2)in the main paper\)\.

## Appendix AMore Analysis

### A\.1Notations

We here give more analysis for the numerical learning\. We first lay out some important notations\. For simplicity, we consider one multi\-digit floating\-point number, which is given by Eq\.[8](https://arxiv.org/html/2605.20369#S4.E8)and we write here again:

𝐠=𝐠0L​𝐠1L−1​⋯​𝐠L0​\.​𝐠L\+2−1​𝐠L\+3−2​⋯​𝐠L\+M\+1−M,\\mathbf\{g\}=\\mathbf\{g\}\_\{0\}^\{L\}\\mathbf\{g\}\_\{1\}^\{L\-1\}\\cdots\\mathbf\{g\}\_\{L\}^\{0\}\\textbf\{\.\}\\mathbf\{g\}\_\{L\+2\}^\{\-1\}\\mathbf\{g\}\_\{L\+3\}^\{\-2\}\\cdots\\mathbf\{g\}\_\{L\+M\+1\}^\{\-M\},\(A\.1\)where the superscripts and the subscripts of𝐠\\mathbf\{g\}represent the digital place and the generation step, respectively\. The total sequenceUUwith lengthTTcan be represented as

U=Up​r​e,𝐱=Up​r​e,x0,x1,⋯,xL,⋯,xL\+M\+1,U=U\_\{pre\},\\mathbf\{x\}=U\_\{pre\},x\_\{0\},x\_\{1\},\\cdots,x\_\{L\},\\cdots,x\_\{L\+M\+1\},\(A\.2\)whereUp​r​eU\_\{pre\}denotes the input sequence preceding the number𝐱\\mathbf\{x\}\. We denotex<tx\_\{<t\}as all preceding input sequences ofxtx\_\{t\}\. The numerical sequence𝐠\\mathbf\{g\}can be decoded from the digital token sequences,

𝐠=Decode​\(𝐱\)\.\\mathbf\{g\}=\\text\{Decode\}\(\\mathbf\{x\}\)\.\(A\.3\)
Tab\.[A\.1](https://arxiv.org/html/2605.20369#A1.T1)presents the notations we used in the analysis\.

### A\.2Analysis to various numerical learning methods

We here give the analysis to various numerical learning methods\. We first simplify the symbols by ignoring the token indexttsince all losses NTL\[[67](https://arxiv.org/html/2605.20369#bib.bib9)\], DIST2Loss and DEL are token\-wised and independent oftt\. We lay out the penalty terms for all the three losses\.

ℛN​T​L=𝐩^​‖Vd−𝐠‖1=∑k𝐩^k​dk,dk=‖k−𝐠‖1,\\displaystyle\\mathcal\{R\}\_\{NTL\}=\\hat\{\\mathbf\{p\}\}\|\|V\_\{d\}\-\\mathbf\{g\}\|\|\_\{1\}=\\sum\\limits\_\{k\}\\hat\{\\mathbf\{p\}\}\_\{k\}d\_\{k\},\\quad d\_\{k\}=\|\|k\-\\mathbf\{g\}\|\|\_\{1\},\(A\.4\)ℛd​i​s​t=ℒKL​\(𝐪∥𝐩^\)=−∑k𝐪k​log⁡𝐩^k−ℋ​\(𝐪\),\\displaystyle\\mathcal\{R\}\_\{dist\}=\\mathcal\{L\}\_\{\\text\{KL\}\}\(\\mathbf\{q\}\\parallel\\hat\{\\mathbf\{p\}\}\)=\-\\sum\\limits\_\{k\}\\mathbf\{q\}\_\{k\}\\log\\hat\{\\mathbf\{p\}\}\_\{k\}\-\\mathcal\{H\}\(\\mathbf\{q\}\),ℛD​E​L=𝐩^​𝐁𝐂𝐄​\(𝐩˙,y\)=∑k𝐩^k​Bk=∑k𝐩^k​\(−yk​log⁡𝐩˙k−\(1−yk\)​log⁡\(1−𝐩˙k\)\)⏟Bk\.\\displaystyle\\mathcal\{R\}\_\{DEL\}=\\hat\{\\mathbf\{p\}\}\\mathbf\{BCE\}\(\\dot\{\\mathbf\{p\}\},y\)=\\sum\\limits\_\{k\}\\hat\{\\mathbf\{p\}\}\_\{k\}B\_\{k\}=\\sum\\limits\_\{k\}\\hat\{\\mathbf\{p\}\}\_\{k\}\\underbrace\{\\Big\(\-y\_\{k\}\\log\\dot\{\\mathbf\{p\}\}\_\{k\}\-\(1\-y\_\{k\}\)\\log\\big\(1\-\\dot\{\\mathbf\{p\}\}\_\{k\}\\big\)\\Big\)\}\_\{B\_\{k\}\}\.The partial derivative of the penalty term w\.r\.t the output logitsz^k\\hat\{z\}\_\{k\}are as follows:

∂ℛN​T​L∂zk=𝐩^k​\(dk−ℛN​T​L\),\\displaystyle\\frac\{\\partial\\mathcal\{R\}\_\{NTL\}\}\{\\partial z\_\{k\}\}=\\hat\{\\mathbf\{p\}\}\_\{k\}\(d\_\{k\}\-\\mathcal\{R\}\_\{NTL\}\),\(A\.5\)∂ℛd​i​s​t∂zk=𝐩^k−𝐪k,\\displaystyle\\frac\{\\partial\\mathcal\{R\}\_\{dist\}\}\{\\partial z\_\{k\}\}=\\hat\{\\mathbf\{p\}\}\_\{k\}\-\\mathbf\{q\}\_\{k\},∂ℛD​E​L∂zk=𝐩^k​\(Bk−ℛD​E​L\+𝐩˙k−yk\)\\displaystyle\\frac\{\\partial\\mathcal\{R\}\_\{DEL\}\}\{\\partial z\_\{k\}\}=\\hat\{\\mathbf\{p\}\}\_\{k\}\(B\_\{k\}\-\\mathcal\{R\}\_\{DEL\}\+\\dot\{\\mathbf\{p\}\}\_\{k\}\-y\_\{k\}\)
###### Proposition 1\.

The optimal solution ofℛD​E​L\\mathcal\{R\}\_\{DEL\}is the one\-hot labelyy\.

###### Proof\.

The one\-hot labelyysatisfiesy𝐠=1y\_\{\\mathbf\{g\}\}=1andyk=0,∀k≠𝐠y\_\{k\}=0,\\,\\forall k\\neq\\mathbf\{g\}\. Since𝐁𝐂𝐄​\(𝐩˙,y\)\\mathbf\{BCE\}\(\\dot\{\\mathbf\{p\}\},y\)has the global minima0when𝐩˙=y\\dot\{\\mathbf\{p\}\}=y, and by nonnegativity of𝐩^\\hat\{\\mathbf\{p\}\}, one can getℛD​E​L\\mathcal\{R\}\_\{DEL\}has the global minima0, which is also one of the global minima of the Shannon entropyℋ​\(𝐩^\)\\mathcal\{H\}\(\\hat\{\\mathbf\{p\}\}\)\. ∎

Analysis to NTL\.We can derive from its partial dericative that NTL encourages to lift up the probabilities for the tokens near the target𝐠\\mathbf\{g\}\. Letsk=dk−ℛN​T​Ls\_\{k\}=d\_\{k\}\-\\mathcal\{R\}\_\{NTL\}, we have the following inequalities,

sk=dk−∑i≠𝐠𝐩^i​di≤dk−∑i≠𝐠𝐩^i=dk−1\+𝐩^𝐠,\\displaystyle s\_\{k\}=d\_\{k\}\-\\sum\\limits\_\{i\\neq\\mathbf\{g\}\}\\hat\{\\mathbf\{p\}\}\_\{i\}d\_\{i\}\\leq d\_\{k\}\-\\sum\\limits\_\{i\\neq\\mathbf\{g\}\}\\hat\{\\mathbf\{p\}\}\_\{i\}=d\_\{k\}\-1\+\\hat\{\\mathbf\{p\}\}\_\{\\mathbf\{g\}\},\(A\.6\)sk=dk−∑i≠𝐠𝐩^i​di≥dk−∑i≠𝐠9​𝐩^i=dk−9\+9​𝐩^𝐠\.\\displaystyle s\_\{k\}=d\_\{k\}\-\\sum\\limits\_\{i\\neq\\mathbf\{g\}\}\\hat\{\\mathbf\{p\}\}\_\{i\}d\_\{i\}\\geq d\_\{k\}\-\\sum\\limits\_\{i\\neq\\mathbf\{g\}\}9\\hat\{\\mathbf\{p\}\}\_\{i\}=d\_\{k\}\-9\+9\\hat\{\\mathbf\{p\}\}\_\{\\mathbf\{g\}\}\.Consequently, we have

dk−9\+9​𝐩^g≤sk≤dk−1\+𝐩^gd\_\{k\}\-9\+9\\hat\{\\mathbf\{p\}\}\_\{g\}\\leq s\_\{k\}\\leq d\_\{k\}\-1\+\\hat\{\\mathbf\{p\}\}\_\{g\}\(A\.7\)This indicates that the logits whose tokens is furthest from𝐠\\mathbf\{g\}with distancedk=9d\_\{k\}=9will be definitely reduced sincesk≥9−9\+9​𝐩^g\>0s\_\{k\}\\geq 9\-9\+9\\hat\{\\mathbf\{p\}\}\_\{g\}\>0and thus the partial derivative∂ℛN​T​L/∂zk\>0\\partial\\mathcal\{R\}\_\{NTL\}/\\partial z\_\{k\}\>0\. Likewise, the logitsz𝐠z\_\{\\mathbf\{g\}\}will be increased sincesk≤0−1\+𝐩^𝐠=𝐩^𝐠−1<0s\_\{k\}\\leq 0\-1\+\\hat\{\\mathbf\{p\}\}\_\{\\mathbf\{g\}\}=\\hat\{\\mathbf\{p\}\}\_\{\\mathbf\{g\}\}\-1<0and thus the partial derivative∂ℛN​T​L/∂zk<0\\partial\\mathcal\{R\}\_\{NTL\}/\\partial z\_\{k\}<0\. For the logits with medium distance, e\.g\.,dk∈\[1,8\]d\_\{k\}\\in\[1,8\], will be optimized according to𝐩^g\\hat\{\\mathbf\{p\}\}\_\{g\}\. It encourages to increase the logits of the token near the target𝐠\\mathbf\{g\}\. When𝐩^𝐠\>8/9\\hat\{\\mathbf\{p\}\}\_\{\\mathbf\{g\}\}\>8/9, indicating the probability of the target token is optimized well, it deducessk\>0s\_\{k\}\>0and all the logits of the non\-target tokens will be further reduced\. To sum up, NTL gradually shifts the probability mass from tokens far from𝐠\\mathbf\{g\}to the tokens near𝐠\\mathbf\{g\}, and finally𝐠\\mathbf\{g\}alone\.

Analysis to DIST2Loss\. By the property of KL divergence, the optimal solution of𝐩^\\hat\{\\mathbf\{p\}\}equals to the soft target𝐪\\mathbf\{q\}\. In this case, one can obtain the digital conditional entropy,

ℋ​\(𝐩^\)=ℋ​\(𝐪\)=−∑k𝐪k​log⁡𝐪k\\mathcal\{H\}\(\\hat\{\\mathbf\{p\}\}\)=\\mathcal\{H\}\(\\mathbf\{q\}\)=\-\\sum\\limits\_\{k\}\\mathbf\{q\}\_\{k\}\\log\\mathbf\{q\}\_\{k\}\(A\.8\)Sinceℋ​\(𝐪\)\\mathcal\{H\}\(\\mathbf\{q\}\)is a constant, it can be directly computed\. There is totally 10 possible ground\-truth digits for𝐠\\mathbf\{g\}\. Due to the symmetry of𝐪\\mathbf\{q\}, it can be reduced to 5 cases\. We list the computation results ofℋ​\(𝐪\)\\mathcal\{H\}\(\\mathbf\{q\}\)in Tab\.[A\.2](https://arxiv.org/html/2605.20369#A1.T2)\. One can see that0\.6456≤ℋ​\(𝐪\)≤1\.07150\.6456\\leq\\mathcal\{H\}\(\\mathbf\{q\}\)\\leq 1\.0715, which is much larger than the token entropy as we have shown in Fig\.[3](https://arxiv.org/html/2605.20369#S5.F3)\. While the cross\-entropy termℋ​\(𝐩,𝐠\)\\mathcal\{H\}\(\\mathbf\{p\},\\mathbf\{g\}\)attempts to optimize𝐩\\mathbf\{p\}to an one\-hot distribution, the penalty term of DIST2Loss still enlarges the digital entropy because the target itself has a large entropy\. This is why we observe the phenomenon of increasing digital entropy from Fig\.[3](https://arxiv.org/html/2605.20369#S5.F3)that the digital entropy of DIST2Loss is approximately 1\.0\.

Table A\.1:Notations\.Table A\.2:The entropy of the distance distribution𝐪\\mathbf\{q\}\.Analysis to Digit Entropy Optimization\. Digit entropy optimization is not new\. For example, it has been independently discussed in\[[5](https://arxiv.org/html/2605.20369#bib.bib66)\]for object detection and in\[[2](https://arxiv.org/html/2605.20369#bib.bib67)\]for LLM reasoning, where entropy minimization is observed to be useful\. Others take the opposite approach: maximum entropy principle\[[28](https://arxiv.org/html/2605.20369#bib.bib68),[60](https://arxiv.org/html/2605.20369#bib.bib56)\]to encourage the models to evolve more diversity of thought chains\. To chase for deterministic number generation, we consider entropy minimization, as presented in Eq\.[6](https://arxiv.org/html/2605.20369#S4.E6)\. In the 5\-th row of Tab\.[A\.3](https://arxiv.org/html/2605.20369#A1.T3), one can see that the overall performance of digit entropy optimization already surpasses previous numerical learning methods, e\.g\., NTL and DIST2Loss\. Due to its unsupervised nature, it is free from the issue of numerical distance\. However, this entropy loss has many trivial solutions\. Any one\-hot distribution can reach the global minima\. Our DEL leverages the digit conditional probability and the binary cross\-entropy, enabling a supervised entropy optimization manner\. The last row of Tab\.[A\.3](https://arxiv.org/html/2605.20369#A1.T3)show that our DEL achieves the best mACC on the 7 benchmarks, demonstrating the effecttiveness of DEL\.

Analysis to Focal Loss\. Adopting binary cross\-entropy loss in multi\-class learning is not new either\. It has been recently demonstrated in\[[27](https://arxiv.org/html/2605.20369#bib.bib83)\]that BCE can maximize the intra\-class compactness and inter\-class distinctiveness among sample features, which makes it has potential to achieve better classification performance than standard cross\-entropy loss\. Early research focal loss\[[29](https://arxiv.org/html/2605.20369#bib.bib82)\]also adopted BCE as the core objectives in object detection\. It was originally proposed to alleviate the gradient dominated problem caused by a large number of negative samples\. This loss function was designed to balance the learning of positive and negative anchor boxes\. Under our defined symbol system, integrating focal loss into numerical learning can be written as

ℒF​L=α​∑kAkγ​Bk=α​∑k\(yk​\(1−𝐩˙\)\+\(1−yk\)​𝐩˙\)γ​\(−yk​log⁡𝐩˙k−\(1−yk\)​log⁡\(1−𝐩˙k\)\)\\mathcal\{L\}\_\{FL\}=\\alpha\\sum\\limits\_\{k\}A\_\{k\}^\{\\gamma\}B\_\{k\}=\\alpha\\sum\\limits\_\{k\}\\Big\(y\_\{k\}\(1\-\\dot\{\\mathbf\{p\}\}\)\+\(1\-y\_\{k\}\)\\dot\{\\mathbf\{p\}\}\\Big\)^\{\\gamma\}\\Big\(\-y\_\{k\}\\log\\dot\{\\mathbf\{p\}\}\_\{k\}\-\(1\-y\_\{k\}\)\\log\\big\(1\-\\dot\{\\mathbf\{p\}\}\_\{k\}\\big\)\\Big\)\(A\.9\)It can be seen that focal loss models each token as a binary classification task on\|V\|\|V\|classes\. Without the softmax\-based probabilities, it is difficult to modeling the inner relationship of digits and is inconsistent with the inference phase\. Moreover, the design principles of focal loss is opposite to our DEL\. Focal loss significantly reduces the loss for easily classified samples by setting the weightAkA\_\{k\}, i\.e\.,Ak→0A\_\{k\}\\rightarrow 0whenyk=1y\_\{k\}=1and𝐩˙k→1\\dot\{\\mathbf\{p\}\}\_\{k\}\\rightarrow 1\. In contrast, DEL focuses more on the high\-confidence tokens\. To support focal loss in LLM training, we make the necessary modifications to enable it to perform in digit subspace\. In Tab\.[A\.3](https://arxiv.org/html/2605.20369#A1.T3), we compare our DEL with the digital focal loss\. The experimental results demonstrate that DEL can achieve better accuracy on 6 of 7 benchmarks\.

Table A\.3:Quantitative comparison between digital focal loss and our DEL\. The base model is Qwen2\.5\-1\.5B\.

## Appendix BHyper\-parameters study

Loss weightα\\alphain DEL\. We investigate the impact of loss function weights on model performance\. Tab\.[C\.4a](https://arxiv.org/html/2605.20369#A3.T4.sf1)reports the quantitative results\. Due to the varying vocabulary sizes of different LLMs, such as 151,936 for Qwen 2\.5, 102,400 for DeepSeek\-Math\-Instruct, 32,016 for CodeLlama, and 32,000 for Mistral, the setting ofα\\alphawill be adjusted accordingly\. We setα=0\.01\\alpha=0\.01for Qwen2\.5 and DeepSeek\-Math\-Instruct andα=0\.1\\alpha=0\.1for CodeLlama and Mistral in other experiments by default\.

kkandβ\\betain floating\-point number optimization\. In Eq\.[9](https://arxiv.org/html/2605.20369#S4.E9), we assign a linear penalty for integers and an exponential decay for decimals\. According to the results in Tab\.[C\.4b](https://arxiv.org/html/2605.20369#A3.T4.sf2), we setk=2k=2by default in all the other experiments\. As for the decimals, we conduct various options\. Tab\.[C\.4c](https://arxiv.org/html/2605.20369#A3.T4.sf3)shows us interesting results that the performance slightly drops if we set no penalty or a constant penalty for the decimals\. What if we treat the decimal as a new integer \(as done by DIST2Loss, see Fig\.[C\.1](https://arxiv.org/html/2605.20369#A3.F1)\), the performance will degrade significantly\. This indicates that the discontinuous penalty between integers and decimals leads to suboptimal results\. In contrast, our floating\-point number optimization can better match the continuity of numerical values, further boosting the performance gains\. We setβ=1\.02\\beta=1\.02by default\.

## Appendix CVisualization

We provide several visual comparison for the numerical learning methods\. We highlight the reasoning errors inredbold\. We use Qwen2\.5\-1\.5B to inference\. In Fig\.[C\.2](https://arxiv.org/html/2605.20369#A3.F2), we show that accurate number prediction is crucial to mathematical reasoning as a small error in number prediction would lead to a complete wrong answer to the mathematical problem\. More examples can be seen in Fig\.[C\.3](https://arxiv.org/html/2605.20369#A3.F3), Fig\.[C\.4](https://arxiv.org/html/2605.20369#A3.F4), Fig\.[C\.5](https://arxiv.org/html/2605.20369#A3.F5)\.

We also present some failure cases that the current numerical learning is hard to handle\. The first one is irrelevant context distraction\[[48](https://arxiv.org/html/2605.20369#bib.bib84)\], as shown in Fig\.[C\.6](https://arxiv.org/html/2605.20369#A3.F6)\. This error is irrelevant to number prediction accuracy, which is simply caused by the distracted attention to the irrelevant contexts\. The second error is calculation error\. The reasoning is correct, however, the calculation results are wrong\. As we have concluded in Sec\.[3](https://arxiv.org/html/2605.20369#S3), numerical reasoning requires language models to possess a range of capabilities, such as logical reasoning, symbolic comprehension, unit conversion, problem decomposition, contextual semantic association\. We believe that in the future, explicitly injecting more capabilities into the optimization process of language models will help improve numerical reasoning\.

Table C\.4:Ablations\. We show ablation experiments for DEL\.\(a\)Loss weightα\\alphain DEL\.
\(b\)Linear weightingkkin integers in floating\-point number optimization\.
\(c\)Exponential decayβ\\betain decimals in floating\-point number optimization\.

\\begin\{overpic\}\[width=433\.62pt\]\{pic/comparison\-digital\-place\} \\put\(24\.5,5\.0\)\{$i$\} \\put\(76\.0,5\.0\)\{$i$\} \\put\(13\.5,2\.0\)\{\(a\) $u\(t\)$ in DIST$\{\}^\{2\}$Loss\} \\put\(66\.0,2\.0\)\{\(b\) $u\(t\)$ in our method\} \\end\{overpic\}Figure C\.1:Illustration of floating\-point optimization\. Our DEL adopts linear penalty for integers \(i≥0i\\geq 0\) and exponential decay in decimals \(i<0i<0\), while previous work DIST2Loss has a discontinuous penalty\. Different colors represent different parameter choices inu​\(t\)u\(t\)\.Question:3​x\+x\+x\+x−3−2=7\+x\+x3x\+x\+x\+x\-3\-2=7\+x\+xIn the equation above, what is the value ofxx?Answer Choices: \(A\)−57\-\\frac\{5\}\{7\}\(B\) 1 \(C\)127\\frac\{12\}\{7\}\(D\) 3Correct: DMLE:We can combine like terms on the left side of the equation to get3​x\+x\+x\+x−3−2=7\+x\+x3x\+x\+x\+x\-3\-2=7\+x\+x4​x−5=7\+2​x\{\\color\[rgb\]\{1,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{1,0,0\}\\textbf\{4\}\}x\-5=7\+2xWe can subtract2​x2xfrom both sides to get2​x−5=72x\-5=7Then we can add 5 to both sides to get2​x=122x=12Finally, we can divide both sides by 2 to getx=6x=6The answer is BNTL:3x\+x\+x\+x\-3\-2=7\+x\+x3x\+x\+x\+x\-5=7\+2x3x\+x\+x\+x\-2x=7\+53x=12x=12/3x=4The answer is BDIST2Loss:3​x\+x\+x\+x−3−2=7\+x\+x3x\+x\+x\+x\-3\-2=7\+x\+x4​x−5=7\+2​x\{\\color\[rgb\]\{1,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{1,0,0\}\\textbf\{4\}\}x\-5=7\+2x2​x=122x=12x=6x=6The answer is DDEL \(Ours\):3x\+x\+x\+x\-3\-2=7\+x\+x3x\+x\+x\+x\-5=7\+2x3x\+x\+x\+x\-2x=7\+53x\+x=124x=12x=3The answer is DFigure C\.2:Visualization of prediction results of LLM\. The reasoning errors are highlighted inredbold\.Question: John plans to sell all his toys and use the money to buy video games\. He has 13 lego sets and he sells them for $15 each\. He ends up buying 8 video games for $20 each and has $5 left\. How many lego sets does he still have?Correct: 2MLE:He earned 13\*15=195 dollars from selling his lego setsHe spent 8\*20=160 dollars on video gamesHe has 195\-160\-5=30 dollars leftHe originally had 30\+160=190 dollarsHe has 190/15=12\.666667 lego sets leftThe answer is 12NTL:He earned 13\*15=195 dollarsHe spent 8\*20=160 dollars on video gamesHe has 195\-160\-5=30 dollars leftHe has 195\-30=165 dollars left to buy lego setsHe has 165/15=11 lego sets leftThe answer is 11DIST2Loss:He earned 13\*15=195 dollars from selling his lego setsHe spent 8\*20=160 dollars on video gamesHe has 195\-160\-5=30 dollars leftHe had 195\-30=165lego setsHe sold 195\-160=35 lego setsHe has 165\-35=130 lego sets leftThe answer is 130DEL \(Ours\):He sold 13 lego sets for 13\*15=195 dollarsHe spent 8\*20=160 dollars on video gamesHe had 195\-160\-5=30 dollars leftHe had 195\-30=165 dollars from selling legosHe had 13\-165/15=13\-11=2 legos leftThe answer is 2Figure C\.3:Visualization of prediction results of LLM\. The reasoning errors are highlighted inredbold\.Question: Charlotte went into the kitchen supply store knowing she wanted a set of pot and pans for $120\.00, a set of mixing bowls for $20\.00 and 5 separate utensils at $5\.00 each\. At check out, the clerk told her everything was 20% off\. How much was her order?Correct: 132MLE:She had a $120\.00 set of pots and pans and a $20\.00 set of mixing bowls so that’s 120\+20 = $140\.00She had 5 separate utensils at $5\.00 each so that’s 5\*5 = $25\.00Her order was $140\.00for pots and pans, $20\.00 for mixing bowls and $25\.00 for utensils for a total of 140\+20\+25 = $185\.00Her order was $185\.00 and it was 20% off so that’s \.20\*185 = $37\.00Her order was $185\.00 and it was 20% off so she paid $37\.00 less so her order was 185\-37 = $148\.00The answer is 148NTL:She had a $20\.00 mixing bowl and 5 separate utensils at $5\.00 each so that’s 20\+5\*5 = $35\.00Her order was $120\.00 for the pots and pans, $20\.00 for the mixing bowls and $35\.00 for the utensils for a total of 120\+20\+35 = $175\.00Her order was $175\.00 and it was 20% off so that’s 175\*\.20 = $35\.00Her order was $175\.00 and it was 20% off so that’s 175\-35 = $140\.00The answer is 140DIST2Loss:She had a $120\.00 set of pots and pans and a $20\.00 set of mixing bowls so that’s 120\+20 = $140\.00She had 5 separate utensils at $5\.00 each so that’s 5\*5 = $25\.00Her order was $140\.00for pots and pans, $20\.00 for mixing bowls and $25\.00 for utensils for a total of 140\+20\+25 = $185\.00Her order was $185\.00 and it was 20% off so that’s 185\*\.20 = $37\.00Her order was $185\.00 and it was 20% off so that’s $185\-$37 = $148\.00The answer is 148DEL \(Ours\):She had a set of pots and pans for $120\.00 and mixing bowls for $20\.00 so her order was $120\.00 \+ $20\.00 = $140\.00She had 5 separate utensils at $5\.00 each so that was 5 \* $5\.00 = $25\.00Her order was $140\.00 \+ $25\.00 = $165\.00Her order was $165\.00 and it was 20% off so that’s 20/100 \* $165\.00 = $33\.00 offHer order was $165\.00 \- $33\.00 = $132\.00The answer is 132Figure C\.4:Visualization of prediction results of LLM\. The reasoning errors are highlighted inredbold\.Question: Valerie earns $5000 per month, 1/2 of what her brother earns\. If their mother earns twice their combined salary, what’s the total amount of money they all have together?Correct: 45000MLE:Valerie earns$5000/2 = $2500per month\.Valerie’s brother earns $2500/2 = $1250 per month\.Their combined salary is $2500 \+ $1250 = $3750 per month\.Their mother earns 2 \* $3750 = $7500 per month\.The total amount of money they all have together is $5000 \+ $3750 \+ $7500 = $16250\.The answer is 16250NTL:Valerie earns 1/2 of what her brother earns, meaning her brother earns 5000\*2=10000 per month\.Together, Valerie and her brother earn 10000\+5000=15000 per month\.Their mother earns twice their combined salary, meaning she earns 2\*15000=30000 per month\.The total amount of money they all have together is 30000\+15000=45000 per month\.The answer is 45000DIST2Loss:Valerie earns$5000/2 = $2500per month\.Her brother earns $2500 \* 2 = $5000 per month\.Their combined salary is $2500 \+ $5000 = $7500 per month\.Their mother earns 2 \* $7500 = $15000 per month\.The total amount of money they all have together is $15000 \+ $7500 = $22500\.The answer is 22500DEL \(Ours\):Valerie’s brother earns 2 \* 5000 = 10000 dollars per month\.Valerie’s mother earns 2 \* \(5000 \+ 10000\) = 30000 dollars per month\.The total amount of money they all have together is 5000 \+ 10000 \+ 30000 = 45000 dollars\.The answer is 45000Figure C\.5:Visualization of prediction results of LLM\. The reasoning errors are highlighted inredbold\.Irrelevant context distractionQuestion:During summer break 644997 kids from Lawrence county stayed home and the other 893835 kids went to camp\. An additional 78 kids from outside the county attended the camp\. How many kids are in Lawrence county?Correct: 1538832Answer: First add the number of kids from Lawrence county who stayed home to the number of kids who went to camp: 644997 \+ 893835 = 1538832Then add the number of kids from outside the county who attended the camp to find the total number of kids in Lawrence county: 1538832 \+ 78 = 1539010The answer is 1539010

Calculation errorQuestion: Question: Michael rented a truck for one day\. There was a rental fee of 20\.99 dollars , and there also was a charge of 25 cents for each mile\. Michael paid 95\.74 dollars when he returned the truck\. For how many miles did he drive the truck?Correct: 299Answer: Let x be the number of miles Michael drove the truck\.The total cost is 20\.99 \+ 0\.25\*x = 95\.74Subtracting 20\.99 from both sides, we get 0\.25\*x = 74\.75Dividing both sides by 0\.25, we get x =303The answer is 303

Figure C\.6:Failure cases\. The reasoning errors are highlighted inredbold\.