entropy

An insightful explainer on the true meaning of entropy, contrasting the common 'disorder' metaphor with a probabilistic interpretation using dice rolling analogies.

A detailed walkthrough explains how Claude Shannon's 1948 information theory underlies LLMs and shows that the 'next-token prediction' story is misleading, linking compression and prediction mathematically.

This paper measures how much formal semantic structure explains human label variation in natural language inference (NLI) using ChaosNLI data, finding group-level effects on entropy but item-level ceilings and null composition effects.

This paper compares semantic search dynamics between humans and LLMs using verbal fluency data, finding that humans exhibit more variable and exploratory search patterns that current models fail to reproduce.

This study uses semantic entropy, an NLP embedding-based metric, to compare semantic memory navigation between blind and sighted individuals. Results show that visual experience influences entropy patterns, with sighted individuals having higher entropy for abstract concepts while blind individuals exhibit higher entropy for visually salient concrete concepts.

The paper identifies 'Thinking Collapse' in on-policy self-distillation for large language models, characterized by a decline in intermediate reasoning steps, and proposes AD-OPSD, a control framework that mitigates this collapse by anchoring high-suppression-risk tokens to a reference prior. The method achieves up to +4.1% absolute average accuracy improvement on mathematical benchmarks.

This paper introduces a self-supervised early exit method for LLMs, allowing computation to stop early at intermediate layers when confidence is high, thereby reducing inference cost. It also presents Dynamic Self-Speculative Decoding (DSSD) which achieves higher token acceptance than existing baselines.

This article evaluates Anthropic's J-Space hallucination detection method across 7 datasets on Qwen3-4B, finding it effective for catching high-confidence errors in factual retrieval but blind to internalized myths and failing on math tasks where thresholds don't transfer.

An intuitive, visual introduction to information theory covering entropy, mutual information, and channel capacity, assuming only basic probability. The paper explains fundamental limits of compression and transmission.

Researchers used a Bose-Einstein condensate as a mini universe to experimentally demonstrate that time can emerge from entropy exchange, supporting the theory of relational or emergent time in quantum physics.

An exploration of why two statistically identical sequences—pure noise and the digits of π—differ in compressibility, distinguishing between statistical redundancy and process-based compression.

This article presents a technique to improve LLM creative writing by modifying the sampling process using entropy, aiming to reduce the generic 'LLM feel' in generated text.

This tweet explores the relationship between statistical mechanics and artificial intelligence, citing a paper that proposes a thermodynamic theory for machine learning systems, introducing concepts like temperature, entropy, and energy, and treating the training process as a phase transition.

Introduces Independent Combinatorial Tokens (ICT) framework that uses Jensen-Shannon divergence between token logit distributions to identify critical branching points, preventing entropy collapse and explosion in RLVR for LLM reasoning. Achieves up to 14.9% pass@4 improvement on Qwen models.

This paper reveals that the clipping mechanism in PPO and GRPO biases entropy in RLVR for LLMs: clip-low increases entropy, clip-high decreases it. The authors prove that standard clipping reduces entropy even with random rewards, and show that adjusting clip-low can prevent entropy collapse and promote exploration.

This is a popular science article of over 25,000 characters, starting from the origin of entropy, reviewing the development of dissipative system theory, and exploring a three-level analysis of whether AI belongs to dissipative systems (hardware level, training level, static model).

This paper proposes Global-Local Uncertainty (GLU), an unsupervised single-pass score that fuses token-level local entropy with hidden-state geometric global entropy for uncertainty quantification in LLMs, showing that the two are near-orthogonal and together capture confident-but-wrong failures.

Bebop proposes entropy-aware multi-token prediction with rejection sampling and a novel TV loss to accelerate RL training of LLMs, achieving up to 1.8x speedup. The method addresses the degradation of acceptance rates during RL by optimizing training objectives.

This paper examines multi-agent systems (MAS) from an entropy perspective, analyzing intra- and inter-agent dynamics. It finds that single agents often outperform MAS and introduces the Entropy Judger algorithm to improve MAS performance.

A technical blog post exploring randomness, Linux entropy, and building a tool called morerandom that uses WASM plugins to feed the system entropy pool.