Revisiting the Uniform Information Density Hypothesis in LLM Reasoning

# Revisiting the Uniform Information Density Hypothesis in LLM Reasoning  
Source: https://arxiv.org/html/2510.06953  

Minju Gwak\(^{1,2}\) Guijin Son\(^{2}\) Jaehyung Kim\(^{1}\)  

\(^{1}\)Yonsei University  
\(^{2}\)OneLine AI  

mjgwak@yonsei\.ac\.kr, jaehyungk@yonsei\.ac\.kr  

###### Abstract  

The Uniform Information Density (UID) hypothesis proposes that effective communication is achieved by maintaining a stable flow of information. In this work, we revisit this principle in the context of Large Language Model (LLM) reasoning, asking whether step-level uniformity reflects reasoning quality. To this end, we introduce a novel framework to quantify uniformity of information flow at both local and global levels, using an entropy-based stepwise density metric. Across experiments on seven reasoning benchmarks, we see a counter-intuitive pattern: while high-quality reasoning exhibits smooth step-by-step transitions (local uniformity) and structured, non-uniform information flow at the trajectory level (global non-uniformity). The results demonstrate that these uniformities outperform alternative internal signals as predictors of reasoning quality, and such divergence with human communication is not a model deficiency, but a byproduct of distinct objectives between human communication and LLM reasoning.  

Code is released at: https://github.com/talzoomanzoo/uid-reasoning  

## 1 Introduction  

Chain-of-Thought (CoT) reasoning has become a central technique for enhancing large language models (LLMs) on complex reasoning tasks (Wei et al., 2023; Kojima et al., 2023; Chae et al., 2023). By generating step-by-step rationales, CoT enables models to decompose problems into simpler subproblems and thereby improve accuracy (Golovneva et al., 2023; Prasad et al., 2023; Yao et al., 2023). Despite these successes, recent studies have highlighted the fragility of this approach (Zhao et al., 2025a). For example, the intermediate rationales are often logically inconsistent or incoherent, and hence models fail to generalize out-of-domain tasks even when producing lengthy reasoning traces (Shojaee et al., 2025). This raises a critical question: how can we determine whether LLMs are reasoning effectively, rather than merely generating superficially coherent text?  

Refer to caption  

Figure 1: Reasoning as information flow. Human communication distributes information smoothly to respect channel capacity, enabling successful understanding. LLM reasoning transmits information across reasoning steps; failures arise from local overload (sharp spikes) and underuse (flat trivial trajectory).  

Clues may lie in human communication itself; the psycholinguistic hypothesis of Uniform Information Density (UID) proposes that speakers distribute information as evenly as possible to balance clarity and efficiency (Fenk and Fenk-Oczlon, 1980; Genzel and Charniak, 2002; Clark et al., 2023; Jaeger and Levy, 2006). Namely, a relatively uniform flow of information is necessary for effective communication (Meister et al., 2021; Aylett and Turk, 2004), aligned with the limits of human cognitive processing. When this balance is disrupted by too much or too little information, communication deteriorates. Motivated by this, we ask whether a similar principle governs reasoning in LLMs.  

As human speakers maintain balanced information flow to support comprehension, effective reasoning traces may require comparable uniformity across steps. Recent findings in cognitive science support this view: Bhambri et al. (2025) shows that reasoning paths interpretable to humans are also easier for models to generate and learn, suggesting a shared structure between human cognition and machine reasoning. To investigate this, we focus on analyzing the information flow of LLM-generated reasoning traces on challenging mathematical benchmarks. Specifically, we begin by defining per-step measurements of information density using entropy of predictive distribution, and examine their relationship to answer correctness. We then introduce two complementary metrics to quantify uniformity at both global and local levels. Our experiments reveal a counter-intuitive pattern: unlike human communication, successful LLM reasoning exhibits high local uniformity but low global uniformity. In our experiments across seven challenging reasoning benchmarks and three LLMs, these uniformities consistently outperform conventional approaches in identifying high-quality trajectories for Best-of-N sampling. We find that this divergence is not a model deficiency, but rather an instrumental byproduct of the distinct objectives between human communication and LLM reasoning. Overall, our contributions are threefold:  

- ∘ To our knowledge, we are the first to revisit the Uniform Information Density (UID) hypothesis in the context of LLM reasoning.  
- ∘ Contrary to our hypothesis, we find that reasoning patterns characterized by global non-uniformity and local uniformity in surprisal correlate with reasoning success on challenging mathematical reasoning tasks.  
- ∘ Extensive analyses show that deviations from such patterns serve as a trace-level internal signal for predicting failure cases, enabling complementary improvements to response-level aggregation and LLM reasoning evaluation.  

## 2 Exploring the Uniform Information Density Hypothesis in LLM Reasoning  

### 2.1 Background: the UID hypothesis  

The UID hypothesis considers language as a signal transmitted through a noisy channel with limited capacity (Meister et al., 2021; Tsipidi et al., 2024). Then, UID posits that speakers aim to convey information efficiently without overwhelming the listener’s processing resources. Formally, let an utterance \(u = [u_{1}, u_{2}, ..., u_{N}]\) be a sequence of \(N\) linguistic units, such as words, subwords, or characters, depending on the granularity of representation. For each unit \(u_{n}\), *surprisal* is defined as its unexpectedness, given its previous context:  
\[
s(u_n) = -\log P(u_n \mid u_{<n}).
\]  
Here, the exponent \(k\) encodes the super-linear nature of processing effort, where rare or unexpected units impose larger effort than predictable ones. Under this formulation, uniform surprisals across units minimize total processing effort, whereas “spiky” linguistic signal with highly uneven surprisal values increase the burden of communication (Meister et al., 2021).  

While UID has been validated in human language, its implications for machine reasoning remain underexplored. LLMs, or more specifically, recent reasoning models such as Deepseek-R1 (DeepSeek-AI et al., 2025) and Qwen3 (Yang et al., 2025a) generate step-by-step CoT traces, similar to how human speech unfolds over time. If we treat each reasoning step \(z_i\) like a unit with surprisal \(s(z_i)\), a single reasoning trace \(z = [z_{1}, z_{2}, ..., z_{N}]\) can be analyzed in the same way to have the total reasoning effort as below:  
\[
E_{\text{reason}}(z) \propto \sum_{n=1}^{N} s(z_n)^k + c \cdot N.
\]  
Then, a natural question arises: does the UID hypothesis hold for good reasoning patterns in LLMs? A smooth, uniform surprisal profile may reflect clear and logical reasoning, while sharp spikes may signal confusion or errors. Therefore, in this work, we validate UID hypothesis beyond psycholinguistics to CoT reasoning of LLMs, offering a new lens on why reasoning models succeed or fail.  

### 2.2 Preliminary analyses: Step-wise information density in CoTs of LLMs  

We start by defining the step-level information density \(ID_i\) for a reasoning trace \(\mathbf{z} = [z_{1},\dots,z_{N}]\), where each reasoning step \(z_i\) is composed of \(M_i\) tokens, i.e., \(z_i = [x_{1},\dots,x_{M_i}]\). We divide the given reasoning trace into multiple reasoning steps using `\n\n`, following Lightman et al. (2023). While we adopt newline-based segmentation, we demonstrate that our findings are robust to alternative stepwise segmentation strategies in Appendix A.  

Let \(p_t\) be the predictive distribution over the vocabulary \(\mathcal{V}\) at the token position \(t\). Then, to characterize \(ID_i\), we consider entropy over tokens in each step:  
\[
H_t = -\sum_{v \in \mathcal{V}} p_t(v) \log p_t(v),
\]  
and step-level information density with entropy is:  
\[
ID_i = \frac{1}{M_i} \sum_{t=1}^{M_i} H_t.
\]  

Refer to caption  

Figure 2: Averaged \(ID_i\) scores of LLM reasoning traces on AIME2025. Correct traces show a downward trend with smooth decay, while incorrect traces show noisy entropy with unresolved spikes.  

#### Justifications for using entropy as a proxy.  
We use entropy as a proxy for information density because it reflects both model confidence and variability in reasoning; low entropy indicates confident predictions while higher entropy implies uncertainty between multiple plausible continuations (Shannon, 1948; Kuhn and Johnson, 2013). Also, in an information-theoretic perspective, entropy quantifies the expected number of bits required to encode the predictive distribution, where higher values corresponding to richer informational content (Cover and Thomas, 2006). Therefore, aggregating entropy across tokens offers a compact and interpretable signal of reasoning difficulty. Furthermore, our experiments (see Appendix C.1) suggest that using entropy as information density is more effective, compared to other candidates such as log-probability and confidence-based methods.  

Figure 2 compares the evolution of \(ID_i\) between the averaged reasoning traces for correct and incorrect solutions in AIME2025, respectively. Here, correct traces exhibit a clear global trend: entropy begins with exploratory fluctuations, stabilizes in mid-trace, and then steadily decays toward near zero, reflecting a structured process of resolution and convergence on the final answer. In contrast, incorrect traces instead display a flat, noisy entropy trajectory with occasional sharp spikes.  

#### Motivation for a structural perspective.  
As we examine entropy peaks at the level of individual reasoning traces, we find that interpreting them solely through the semantic content of the text is difficult (see Appendix C.2). To be specific, (1) entropy levels in both correct and incorrect traces can be quite varying, (2) number of transition words (i.e., *But*, *Alternatively*, *Wait*) may appear more at correct traces contrary to our intuition, and (3) wrong reasoning traces may be concise and have fewer number of steps. More importantly, such an approach does not provide a consistent basis for interpretation across various benchmarks, highlighting the need for a unified structural perspective.  

Building on such motivation, we introduce a framework for measuring the uniformity of information density in reasoning traces.  

### 2.3 Measuring global and local uniformity of information density in CoTs of LLMs  

To quantify the uniformity of information density in a reasoning trace, we first distinguish between two complementary notions of uniformity that have been discussed in prior psycholinguistic work (Meister et al., 2021; Collins, 2014). **Global uniformity** characterizes whether information is distributed evenly across the entire trace, corresponding to a relatively stable surprisal level over long horizons. In contrast, **local uniformity** captures whether information changes smoothly between adjacent steps, reflecting gradual and coherent transitions rather than abrupt jumps. These two notions capture uniformity in different ways and therefore diverge substantially in reasoning traces: a trace may appear globally uniform yet contain sharp local disruptions, or conversely, exhibit smooth local transitions while concentrating information unevenly across steps.  

Motivated by this distinction, we introduce two complementary UID-based metrics that separately operationalize global and local uniformity in LLM reasoning traces: (1) *global variance* and (2) *local step-to-step spikes and falls*.  

#### Global uniformity via variance.  
Global uniformity is measured by the variance of step-level information density across the entire reasoning trace, capturing whether the information is evenly distributed or concentrated in a small number of steps. Formally, for a reasoning trace \(\mathbf{z} = [z_{1},\dots,z_{N}]\), let us define the non-negative information density vector \(\mathbf{u} = [ID_{1},\dots,ID_{N}]\). After min–max normalization, where \(ID_i' = \frac{ID_i - m}{M - m}\) with \(m = \min_{1 \le i \le N} ID_i\) and \(M = \max_{1 \le i \le N} ID_i\), we obtain the normalized vector \(\tilde{\mathbf{u}} = [ID_{1}',\dots,ID_{N}']\). The variance of the normalized information density values is then defined as:  
\[
\mathrm{Var}(\tilde{\mathbf{u}}) = \frac{1}{N} \sum_{i=1}^{N} \left( ID_i' - \mu \right)^2,
\]  
where \(\mu = \frac{1}{N} \sum_{i=1}^{N} ID_i'\). High variance indicates global non-uniformity, where information is unevenly concentrated across steps, while lower variance corresponds to globally uniform traces.  

#### Local uniformity via step-to-step smoothness.  
Local uniformity captures how smoothly information density evolves between adjacent reasoning steps, measuring whether uncertainty is resolved gradually or through abrupt transitions. Given \(\tilde{\mathbf{u}}\), we define the step-to-step change as \(\Delta_i = ID_i' - ID_{i-1}'\) for \(i=2,\dots,N\), and compute the mean and standard deviation of the change sequence as \(\mu_{\Delta} = \frac{1}{N-1} \sum_{i=2}^{N} \Delta_i\) and \(\sigma_{\Delta} = \sqrt{\frac{1}{N-1} \sum_{i=2}^{N} (\Delta_i - \mu_{\Delta})^2}\). To identify significant local disruptions, we define thresholds \(T^{+} = \mu_{\Delta} + \tau \sigma_{\Delta}\) and \(T^{-} = \mu_{\Delta} - \tau \sigma_{\Delta}\), where \(\tau \in \{2,3\}\). Then, an *upward spike* is identified when \(\Delta_i > T^{+}\)