小型语言模型知道自己不知道什么吗？

# Can Small Language Models Know What They Don’t Know? Semantic Entropy as a Confidence Signal for Sub-3B Parameter Models
Source: [https://arxiv.org/html/2609.20824](https://arxiv.org/html/2609.20824)
###### Abstract

We explore whether entropy\-based confidence signals can be leveraged to improve the accuracy of Small Language Models \(SLMs\) with fewer than 3 billion parameters, running entirely on consumer hardware\. We evaluate seven distinct approaches, including token\-level entropy early stopping, semantic entropy estimation, and uncertainty\-aware routing to larger expert models, across 7 model pairs and 5 standard NLU benchmarks\. Our key finding is thattoken\-level entropy is effectively blind in SLMs: in 91% of dataset–model combinations, mean token entropy is near zero regardless of answer correctness, rendering token\-based confidence signals unusable at this scale\. We demonstrate thatsemantic entropy, computed by generating multiple samples, clustering answers by meaning, and measuring distributional uncertainty, recovers a viable confidence signal\. Using semantic entropy to selectively route uncertain queries to a larger expert model yields accuracy improvements of up to \+50 percentage points\. Notably, cross\-family routing \(e\.g\., SmolLM 360M to Phi\-3\.5\-mini\) averages \+22\.0% improvement compared to \+6\.8% for same\-family routing, revealing that expert model quality matters more than architectural compatibility\. Our results suggest that the value proposition for entropy\-based methods in SLMs is not computational savings but*intelligent compute allocation*: spending more tokens where they matter most\.

## 1Introduction

Large Language Models \(LLMs\) with tens or hundreds of billions of parameters have demonstrated remarkable reasoning capabilities, and Shannon entropy computed from token\-level log\-probabilities has been explored as a confidence signal for guiding inference decisions\[[1](https://arxiv.org/html/2609.20824#bib.bib1),[2](https://arxiv.org/html/2609.20824#bib.bib2)\]\. However, the practical deployment landscape increasingly demands capable models that can run on consumer\-grade hardware such as laptops, edge devices, and single\-GPU workstations, where models are constrained to fewer than 7 to 8 billion parameters\.

This raises a natural question:do entropy\-based confidence signals transfer to Small Language Models?

We set out to answer this question through an exploratory study\. Rather than proposing a single method and evaluating it, we systematically compare seven different confidence\-based approaches across diverse model families and scales, seeking to understand what works, what fails, and why\.

Our investigation yields three principal findings:

1. 1\.Token\-level entropy is blind in SLMs\.In 32 out of 35 dataset–model combinations \(91%\), the mean token entropy is effectively zero \(<0\.01<0\.01bits\)\. SLMs produce short, maximally confident outputs regardless of correctness\. Token\-level confidence signals do not emerge at this scale\.
2. 2\.Semantic entropy recovers a useful uncertainty signal\.By generating multiple samples and measuring disagreement at the*answer level*rather than the*token level*, we can identify when an SLM is uncertain\. This signal enables selective routing to a larger expert model, improving accuracy by up to \+50 percentage points\.
3. 3\.Expert quality matters more than architectural compatibility\.Cross\-family routing to a high\-quality expert \(e\.g\., SmolLM 360M to Phi\-3\.5\-mini\) yields substantially larger gains \(\+26\.0% average\) than same\-family routing to a weaker sibling model \(\+3\.0% average\)\. The routing mechanism itself is not the bottleneck; the expert’s absolute capability is\.

Importantly, these accuracy gains come at the cost of*more*tokens, not fewer: semantic approaches require generating multiple samples, and routing invokes a second, larger model\. In all 35 dataset–model combinations, semantic routing used more tokens than the baseline\. The value proposition shifts from “spend less” to “spend smarter,” allocating additional compute precisely where the small model is uncertain\.

![Refer to caption](https://arxiv.org/html/2609.20824v1/token_entropy_blind.png)Figure 1:Token\-level entropy is blind in SLMs\.Left:Mean and maximum token entropy aggregated by model family across all 5 datasets\. Three of four families produce entropy below 0\.001 bits; only Llama\-1B shows any measurable variation \(max 0\.026 bits\), still well below a useful discrimination threshold\.Right:Correct and incorrect answers produce identical near\-zero mean token entropy, making threshold\-based confidence discrimination impossible at this scale\.
## 2Background

### 2\.1Shannon Entropy as a Confidence Signal

Shannon entropy\[[3](https://arxiv.org/html/2609.20824#bib.bib3)\]quantifies the uncertainty in a probability distribution\. For a language model generating tokenttwith a distribution over the vocabulary, the entropy is:

H=−∑i=1kpi​log2⁡piH=\-\\sum\_\{i=1\}^\{k\}p\_\{i\}\\log\_\{2\}p\_\{i\}\(1\)wherepip\_\{i\}are the probabilities of the top\-kktokens after softmax normalization\. A lower entropy indicates higher confidence \(the model concentrates probability mass on fewer tokens\), while higher entropy indicates uncertainty \(probability is spread across many tokens\)\.

For a generated sequence ofTTtokens, the mean entropy provides an aggregate confidence signal:

Hmean=1T​∑t=1THtH\_\{\\text\{mean\}\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}H\_\{t\}\(2\)
This signal has been explored for guiding inference in large reasoning models\[[1](https://arxiv.org/html/2609.20824#bib.bib1),[2](https://arxiv.org/html/2609.20824#bib.bib2)\], but its behavior in small models \(<<4B parameters\) remains largely unexplored\.

### 2\.2Semantic Entropy

Semantic entropy\[[4](https://arxiv.org/html/2609.20824#bib.bib4)\]addresses a fundamental limitation of token\-level entropy: a model may be confident about each individual token it produces while being uncertain about the overall*meaning*of its answer\. Semantic entropy operates at the answer level:

1. 1\.GenerateNNindependent samples from the model \(using temperature sampling\)
2. 2\.Cluster the answers by semantic meaning
3. 3\.Compute Shannon entropy over the cluster probability distribution

ForCCmeaning clusters with probabilities\{q1,q2,…,qC\}\\\{q\_\{1\},q\_\{2\},\\ldots,q\_\{C\}\\\}:

Hsemantic=−∑c=1Cqc​log2⁡qcH\_\{\\text\{semantic\}\}=\-\\sum\_\{c=1\}^\{C\}q\_\{c\}\\log\_\{2\}q\_\{c\}\(3\)
When all samples agree \(1 cluster\),Hsemantic=0H\_\{\\text\{semantic\}\}=0\. When samples are maximally scattered \(NNclusters forNNsamples\), entropy reacheslog2⁡N\\log\_\{2\}N\. ForN=5N=5samples, the semantic entropy ranges from 0\.0 to 2\.32 bits\.

### 2\.3Model Cascading

Model cascading \(or routing\) is a strategy where queries are first processed by a smaller, cheaper model, and selectively forwarded to a larger, more capable model when the smaller model’s confidence is low\[[5](https://arxiv.org/html/2609.20824#bib.bib5)\]\. Our work uses entropy\-based confidence as the routing signal\.

## 3Methodology

### 3\.1Approaches Evaluated

We evaluate seven approaches that combine different confidence signals with different actions:

1. 1\.Baseline: Generate a single response from the small model with no optimization\. This represents the default inference behavior\.
2. 2\.Token Entropy Early Stop: Compute the mean Shannon entropy across generated tokens\. If entropy falls below a calibrated thresholdτ\\tau, accept the short response\. Otherwise, allow the model to continue generating up to the maximum token limit\.
3. 3\.MaxProb Early Stop: Use the mean maximum token probability \(highest softmax probability per position\) as the confidence signal instead of entropy\.
4. 4\.ProbGap Early Stop: Use the mean probability gap \(p1−p2p\_\{1\}\-p\_\{2\}, difference between the top\-two token probabilities\) as the confidence signal\.
5. 5\.Token Entropy Routing: Compute token\-level entropy\. If entropy exceeds the threshold \(indicating low confidence\), route the query to a larger expert model instead of using the small model’s answer\.
6. 6\.Semantic Entropy Early Stop: GenerateN=5N=5samples, compute semantic entropy over answer clusters\. If semantic entropy is below the threshold, accept the dominant answer\. Otherwise, continue with the full generation\.
7. 7\.Semantic Entropy Routing: GenerateN=5N=5samples and compute semantic entropy\. If semantic entropy exceeds the threshold, route the query to a larger expert model\. This combines semantic uncertainty detection with model cascading \(Figure[2](https://arxiv.org/html/2609.20824#S3.F2)\)\.

### 3\.2Threshold Calibration

For each approach, we calibrate the confidence thresholdτ\\tauusing a held\-out calibration set of 15 examples per dataset\. The threshold is set to the mean entropy \(or confidence metric\) of examples where the small model produces correct answers:

τ=1\|𝒞\|​∑x∈𝒞H​\(x\)\\tau=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{x\\in\\mathcal\{C\}\}H\(x\)\(4\)where𝒞\\mathcal\{C\}is the set of correctly answered calibration examples\. This ensures that the threshold captures the typical confidence level when the model “knows” the answer\.

### 3\.3Answer Clustering for Semantic Entropy

For multiple\-choice questions \(MCQ\), we extract the answer letter \(A/B/C/D\) and cluster by exact match\. For boolean questions, we extract yes/no\. For free\-text answers, we use bag\-of\-tokens embeddings with agglomerative clustering \(cosine similarity, average linkage, threshold 0\.85\)\.

### 3\.4Framework Architecture

Algorithm 1Semantic Entropy Routing1:Query

qq, Small model

MsM\_\{s\}, Expert model

MeM\_\{e\}, Threshold

τ\\tau, Number of samples

NN
2:Generate

NNsamples:

\{a1,…,aN\}←Ms​\(q\)\\\{a\_\{1\},\\ldots,a\_\{N\}\\\}\\leftarrow M\_\{s\}\(q\)
3:Extractcore answers from each sample

4:Clusteranswers by meaning

→C\\rightarrow Cclusters

5:Compute

Hsemantic=−∑c=1Cqc​log2⁡qcH\_\{\\text\{semantic\}\}=\-\\sum\_\{c=1\}^\{C\}q\_\{c\}\\log\_\{2\}q\_\{c\}
6:if

Hsemantic\>τH\_\{\\text\{semantic\}\}\>\\tauthen

7:Route:

answer←Me​\(q\)\\text\{answer\}\\leftarrow M\_\{e\}\(q\)⊳\\trianglerightUse expert

8:else

9:Accept:

answer←dominant cluster answer\\text\{answer\}\\leftarrow\\text\{dominant cluster answer\}
10:endif

11:returnanswer

![Refer to caption](https://arxiv.org/html/2609.20824v1/framework_overview.jpg)Figure 2:Semantic entropy routing framework\.Given an input query, the SLM generatesN=5N=5independent samples via temperature sampling\. Answers are clustered by meaning \(exact match for MCQ; agglomerative clustering for free\-text\)\. Shannon entropy is computed over the cluster distribution\. If semantic entropy exceeds the calibrated thresholdτ\\tau\(set to the mean entropy of correctly answered calibration examples\), the query is routed to a larger expert model\. Otherwise, the SLM’s dominant\-cluster answer is accepted\.

## 4Experimental Setup

### 4\.1Models

We evaluate 7 model pairs spanning 4 model families\. Table[1](https://arxiv.org/html/2609.20824#S4.T1)summarizes the configurations\. All experiments run on a single consumer machine \(Apple Silicon\) without quantization, demonstrating feasibility on commodity hardware\.

Table 1:Model pairs evaluated\. “Cross” denotes cross\-family routing where small and expert models are from different architectures\.
### 4\.2Datasets

We evaluate on 5 standard NLU benchmarks covering different reasoning capabilities:

- •BoolQ\[[6](https://arxiv.org/html/2609.20824#bib.bib6)\]: Boolean question answering \(yes/no\)
- •HellaSwag\[[7](https://arxiv.org/html/2609.20824#bib.bib7)\]: Commonsense sentence completion \(4\-way MCQ\)
- •ARC\-Challenge\[[8](https://arxiv.org/html/2609.20824#bib.bib8)\]: Grade\-school science reasoning \(4\-way MCQ\)
- •ARC\-Easy\[[8](https://arxiv.org/html/2609.20824#bib.bib8)\]: Easier subset of ARC \(4\-way MCQ\)
- •WinoGrande\[[9](https://arxiv.org/html/2609.20824#bib.bib9)\]: Commonsense coreference resolution \(2\-way\)

We use 50 examples per dataset for most model pairs, with 15 held\-out examples for threshold calibration\.

### 4\.3Configuration

All experiments use temperatureT=0\.7T=0\.7, top\-ppsampling withp=0\.95p=0\.95, maximum generation length of 128 tokens, and top\-k=20k=20logits for entropy computation\. Semantic entropy approaches generateN=5N=5samples per query\. Figure[3](https://arxiv.org/html/2609.20824#S4.F3)summarizes the full experimental coverage\.

![Refer to caption](https://arxiv.org/html/2609.20824v1/experimental_design.png)Figure 3:Experimental coverage\.Each cell represents one dataset–model pair evaluated across all 7 approaches\. Darker rows indicate cross\-family configurations, where the expert is from a different architecture \(Phi\-3\.5\-mini\)\. Sample sizes range from 20 to 50 per cell, with 15 held\-out calibration examples per dataset\.

## 5Results

### 5\.1Token Entropy is Blind in SLMs

Our most striking finding is that token\-level entropy provides no useful signal for small language models\. Table[2](https://arxiv.org/html/2609.20824#S5.T2)shows the mean token entropy across all model pairs and datasets\.

Table 2:Mean token entropy \(bits\) for baseline generation\. Values below 0\.01 are effectively zero\. 32 out of 35 combinations \(91%\) show near\-zero entropy\.Figure[1](https://arxiv.org/html/2609.20824#S1.F1)visualizes this collapse\. SLMs produce extremely short, high\-confidence outputs where every token has probability\>\>0\.999, yielding entropy indistinguishable from zero\. Both correct and incorrect answers exhibit this same near\-zero entropy, making threshold\-based discrimination impossible: the calibrated threshold itself converges to zero\.

This phenomenon arises because SLMs, when faced with questions they cannot reason through, tend to emit short, decisive answers \(“A”, “Yes”, “No”\) rather than extended reasoning chains\. Each token in these short responses is produced with maximal confidence\.

### 5\.2Semantic Entropy Recovers a Useful Signal

While SLMs are token\-level confident, they are*not*semantically consistent\. When prompted multiple times with temperature sampling, an uncertain SLM produces different answers across runs\. Table[3](https://arxiv.org/html/2609.20824#S5.T3)presents the semantic routing results for all model pairs\.

Table 3:Accuracy \(%\) across all approaches for each model pair and dataset\. Best result per row inbold\.Δ\\Deltashows semantic routing improvement over baseline\.Semantic routing wins the plurality of comparisons: it achieves the best accuracy in 13 out of 35 dataset–model pairs \(37%\), followed by baseline at 11/35 \(31%\), token early stop at 5/35 \(14%\), token routing at 4/35 \(11%\), and semantic early stop at 2/35 \(6%\)\.

Figure[4](https://arxiv.org/html/2609.20824#S5.F4)shows the accuracy comparison across all approaches for our strongest result \(SmolLM 360M to Phi\-3\.5\-mini\), and Figure[5](https://arxiv.org/html/2609.20824#S5.F5)shows the semantic routing mechanism in detail for this pair, illustrating both the accuracy improvement and the fraction of queries routed to the expert model per dataset\.

![Refer to caption](https://arxiv.org/html/2609.20824v1/acc_smol_to_phi.png)Figure 4:Accuracy across all approaches for SmolLM 360M to Phi\-3\.5\-mini \(cross\-family routing to the best expert\)\. Semantic Routing \(rightmost bars\) achieves the largest gains on reasoning\-heavy datasets, with \+50 percentage points over baseline on ARC\-Challenge and ARC\-Easy\.![Refer to caption](https://arxiv.org/html/2609.20824v1/semrt_smol_to_phi.png)Figure 5:Semantic routing analysis for SmolLM 360M to Phi\-3\.5\-mini\.Left:Accuracy improvement over baseline, reaching \+50% on ARC\-Challenge and ARC\-Easy\.Right:Routing rate, i\.e\., the fraction of queries where semantic entropy exceeded the threshold and the query was forwarded to the expert model\. HellaSwag routes 80% of queries, corresponding to the dataset where the small model is most uncertain\.
### 5\.3The Role of Expert Model Quality

Perhaps our most practically significant finding is the dramatic difference between same\-family and cross\-family routing\. Table[4](https://arxiv.org/html/2609.20824#S5.T4)summarizes the average accuracy improvement from semantic routing across model pair types\.

Table 4:Average accuracy improvement \(Δ\\Delta%\) from semantic routing over baseline, grouped by model pair\.The contrast is striking: SmolLM 360M routed to its own larger sibling \(1\.7B\) achieves only \+3\.0% average improvement, but the*same model*routed to Phi\-3\.5\-mini achieves \+26\.0%\. The routing mechanism and uncertainty signal are identical; the only difference is the expert’s quality\. This suggests that deploying a single high\-quality expert as a shared “fallback” for multiple weak SLMs is more effective than pairing each SLM with its family’s larger variant\. Figure[6](https://arxiv.org/html/2609.20824#S5.F6)visualizes this finding\.

![Refer to caption](https://arxiv.org/html/2609.20824v1/cross_vs_same.png)Figure 6:Average accuracy improvement from semantic routing across all 7 model pairs\. Cross\-family routing \(blue\) to a high\-quality expert averages \+22\.0% compared to \+6\.8% for same\-family routing \(green\)\. Dashed lines show group averages\.The Gemma3 results represent an instructive failure case: Gemma3\-1B produces near\-zero semantic entropy across all datasets \(the model gives consistent but incorrect answers\), so the routing signal never fires\. This suggests that some model families may produce overconfident outputs even at the semantic level\.

### 5\.4Token Usage: Spending More, Not Less

Semantic routing in SLMs*always*uses more tokens than baseline\. In all 35 dataset–model combinations, semantic routing consumed more compute, typically around 640 tokens per query compared to 2–120 tokens for baseline generation\.

This increased cost comes from two sources: \(1\) generatingN=5N=5samples for semantic entropy estimation, and \(2\) invoking the expert model for routed queries\. In a typical configuration \(Qwen 0\.5B to 3B\), baseline generation uses 30–120 tokens per query, while semantic routing consumes∼\\sim640 tokens\. The value proposition is not efficiency but accuracy: the additional compute is allocated precisely where the small model is uncertain, yielding targeted accuracy improvements\.

## 6Analysis

### 6\.1Why Token Entropy Fails in SLMs

The failure of token\-level entropy in SLMs can be traced to a fundamental behavioral pattern\. Small models tend to:

- •Generate very short responses \(often 2–5 tokens for MCQ tasks\)
- •Assign near\-unity probability to each token in the sequence
- •Skip reasoning entirely, jumping directly to an answer

As a result, even when the answer is incorrect, each token is produced with probability\>\>0\.999, yielding entropy<<0\.001 bits\. The calibrated threshold converges to∼\\sim0\.0, eliminating any discriminative power\.

### 6\.2How Semantic Entropy Detects Uncertainty

Semantic entropy circumvents this failure mode by probing the model’s uncertainty at a different level\. Consider a concrete example from our experiments \(Qwen 0\.5B on HellaSwag, correct answer: B\):

- •Token entropy: The model generates “D” with token entropy = 0\.000 bits\. The confidence signal suggests maximum certainty\.
- •Semantic entropy: Five samples yield answers \{D, A, B, D, C\}, producing four distinct clusters with probabilities\{0\.4,0\.2,0\.2,0\.2\}\\\{0\.4,0\.2,0\.2,0\.2\\\}, givingHsemantic=1\.922H\_\{\\text\{semantic\}\}=1\.922bits\. This exceeds the threshold \(1\.343 bits\), triggering routing to Qwen 3B, which correctly answers “B”\.

The key insight is that while the model is confident about each*token*it produces, repeated sampling reveals that it is*not*confident about which*answer*is correct\. The model “knows what it doesn’t know” at the answer level, even when it appears confident at the token level\.

### 6\.3Factors Affecting Routing Effectiveness

We observe that routing effectiveness depends on several factors:

Expert model capability\.The absolute quality of the expert model is the strongest predictor of routing success\. Phi\-3\.5\-mini, known for punching above its weight class, consistently yields the largest improvements as an expert\.

Small model uncertainty behavior\.Some models \(e\.g\., Gemma3\-1B\) produce semantically consistent but incorrect answers, yielding low semantic entropy and preventing the routing signal from firing\. Effective routing requires that the small model’s uncertainty manifests as answer variation across samples\.

Dataset characteristics\.Routing benefits are largest for reasoning\-heavy tasks \(HellaSwag, ARC\) where the expert model’s additional capability translates to correctness\. For simpler binary tasks \(BoolQ\) where the small model already performs well, routing provides minimal benefit\.

Figure[7](https://arxiv.org/html/2609.20824#S6.F7)contrasts the accuracy change heatmaps for a successful model pair \(Qwen 0\.5B to 3B\) and a failure case \(Gemma3 1B to 4B\), highlighting how semantic routing effectiveness varies with model characteristics\.

![Refer to caption](https://arxiv.org/html/2609.20824v1/heatmap_qwen.png)\(a\)Qwen 0\.5B to 3B: Semantic routing shows strong gains \(dark green\) across HellaSwag, ARC\-Challenge, and ARC\-Easy\.
![Refer to caption](https://arxiv.org/html/2609.20824v1/heatmap_gemma3.png)\(b\)Gemma3 1B to 4B: No approach improves over baseline\. Gemma3\-1B produces overconfident outputs even at the semantic level\.

Figure 7:Accuracy change vs\. baseline heatmaps comparing a successful routing pair \(left\) with a failure case \(right\)\. Green indicates improvement, orange/red indicates degradation\.
### 6\.4Illustrative Examples

Table[5](https://arxiv.org/html/2609.20824#S6.T5)presents representative examples from our experiments showing the routing decision process in action\.

Table 5:Routing examples \(Qwen 0\.5B to 3B\)\. Token entropy is 0\.000 in all cases; semantic entropy correctly identifies uncertainty and triggers routing\.Semantic entropy values\(N=5N\\\!=\\\!5, exact\-match clustering\): 5\-0\-0\-0 \(unanimous\) = 0\.0002\-2\-1 \(contested\) = 1\.5224\-1 \(one outlier\) = 0\.7222\-1\-1\-1 \(scattered\) = 1\.9223\-2 \(majority split\) = 0\.9711\-1\-1\-1\-1 \(all differ\) = 2\.3223\-1\-1 \(fragmented\) = 1\.371

## 7Discussion

### 7\.1Limitations

Sample size\.Our experiments use 20–50 examples per dataset–model combination\. While sufficient for identifying trends and large effects, statistical significance of smaller differences \(<<10 percentage points\) cannot be established at this scale\. We present our findings as exploratory observations rather than statistically confirmed results\.

Compute overhead\.Semantic entropy requiresN=5N=5forward passes through the small model plus \(for routed queries\) one pass through the expert model\. For a 360M\-parameter model routing to a 3\.8B expert, this represents a∼\\sim6×\\timesincrease in total compute per query\. The accuracy gains must be weighed against this cost in practical deployment\.

Model coverage\.Our study covers 4 model families \(Qwen, Llama, SmolLM, Gemma\) and 2 cross\-family expert configurations\. The Gemma3 failure case suggests that some architectures may not exhibit exploitable semantic uncertainty, and broader model coverage would help delineate when semantic routing is applicable\.

Clustering method\.Our bag\-of\-tokens embedding approach for free\-text clustering is a simple baseline\. More sophisticated methods \(e\.g\., sentence\-transformers\) might yield better semantic grouping for non\-MCQ tasks\.

### 7\.2Practical Implications

Our findings suggest a practical deployment architecture for resource\-constrained environments: run a small, fast model \(<<1B parameters\) for all queries, and maintain a single high\-quality expert \(e\.g\., Phi\-3\.5\-mini at 3\.8B\) as a shared fallback\. Semantic entropy provides the routing signal, and the expert model need not be from the same family as the small model\. This architecture enables:

- •Low latency for easy queries: Most queries are answered by the fast small model without routing\.
- •Accuracy recovery for hard queries: Uncertain queries are identified via semantic entropy and routed to the expert\.
- •Cost control: The routing rate \(what fraction of queries go to the expert\) is controlled by the thresholdτ\\tau, allowing operators to trade accuracy for compute cost\.

### 7\.3Relationship to Prior Work

Entropy\-based confidence estimation has been studied in the context of larger models\[[1](https://arxiv.org/html/2609.20824#bib.bib1),[2](https://arxiv.org/html/2609.20824#bib.bib2)\]\. Our work investigates whether these signals transfer to small models and finds that token\-level entropy collapses, requiring a shift to semantic\-level measurement\. Semantic entropy\[[4](https://arxiv.org/html/2609.20824#bib.bib4)\]was originally proposed for uncertainty estimation; we demonstrate its utility as a*routing signal*for small models, connecting it to model cascading\[[5](https://arxiv.org/html/2609.20824#bib.bib5)\]\. Our use of semantic entropy as the cascading criterion is, to our knowledge, new\.

## 8Conclusion

We investigated whether entropy\-based confidence signals can help Small Language Models \(<<3B parameters\) know what they don’t know\. Our exploratory study across 7 model pairs, 5 datasets, and 7 approaches reveals that token\-level entropy is blind in SLMs, with 91% of test configurations showing near\-zero entropy regardless of correctness\. Semantic entropy, which measures answer\-level disagreement across multiple samples, recovers a viable uncertainty signal that enables effective routing to larger expert models, with accuracy improvements of up to \+50 percentage points\. The expert’s quality matters more than family match: cross\-family routing to a capable expert averages \+22% improvement versus \+6\.8% for same\-family routing\.

The value proposition for entropy\-based methods in SLMs is not about saving tokens; it is about spending them wisely\. By identifying precisely when a small model is uncertain and selectively invoking a stronger model, we transform a limitation \(the small model’s ignorance\) into an actionable signal for intelligent compute allocation\.

## References

- \[1\]A\. Sharma and P\. Chopra\.Think Just Enough: Sequence\-Level Entropy as a Confidence Signal for LLM Reasoning\.*arXiv preprint arXiv:2510\.08146*, 2025\.
- \[2\]J\. Han, C\. Gong, Z\. Chen, S\. Liu, B\. Zheng, and L\. Feng\.Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking\.*arXiv preprint arXiv:2509\.23392*, 2025\.
- \[3\]C\. E\. Shannon\.A Mathematical Theory of Communication\.*Bell System Technical Journal*, 27\(3\):379–423, 1948\.
- \[4\]L\. Kuhn, Y\. Gal, and S\. Farquhar\.Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation\.In*International Conference on Learning Representations*, 2023\.
- \[5\]L\. Chen, M\. Zaharia, and J\. Zou\.FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance\.*arXiv preprint arXiv:2305\.05176*, 2023\.
- \[6\]C\. Clark, K\. Lee, M\.\-W\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova\.BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions\.In*NAACL*, 2019\.
- \[7\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\.HellaSwag: Can a Machine Really Finish Your Sentence?In*ACL*, 2019\.
- \[8\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\.Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge\.*arXiv preprint arXiv:1803\.05457*, 2018\.
- \[9\]K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi\.WinoGrande: An Adversarial Winograd Schema Challenge at Scale\.In*AAAI*, 2020\.