entropy

This paper introduces TAM-Chain, a multi-scale thyroid cytology classification method that employs absorbing Markov chains and Shannon entropy for uncertainty quantification, aiming to suppress false negatives and adapt to domain shifts.

FLEET introduces a memory mechanism to text generation in large language models, using logits entropy to enhance trajectories, resulting in improved accuracy and a 3x speedup, particularly on coding tasks.

This preregistered reproduction study validates that the shape of chain-of-thought entropy trajectories predicts large language model answer correctness, while the total entropy drop is inconsistent across settings, and explores final-step entropy as an improved metric.

This article explores using Byte-Pair Encoding (BPE) token efficiency as a more effective alternative to entropy for detecting secrets in code, focusing on statistical rarity over randomness.

This paper introduces 'Agentic Pressure' as an endogenous force that destabilizes AI agent safety when compliance conflicts with goal achievement, proposing a formal framework validated by empirical experiments.

This paper challenges the standard assumptions about the linguistic units of the Voynich manuscript, showing through quantitative analysis that glyphs, tokens, and spaces do not correspond to letters, words, and word spaces as commonly believed.

The paper presents SAGE, a method that adapts surrogate gradients for Spiking Transformers using attention-derived entropy to improve training accuracy, demonstrated on CIFAR-10/100 datasets.

The article discusses the mathematical equivalence between compression and prediction in information theory, referencing classical foundations and modern applications like language models.

This paper identifies a privacy vulnerability in RL-trained multimodal large reasoning models, which can leak sensitive facts in their reasoning traces even after unlearning, and proposes LEMUR, a training-free inference-time framework that uses entropy dynamics to detect and suppress such leakage.

This paper introduces Lodestar, a method that uses reinforcement learning to train a short polarizer prompt string that helps a frozen LLM avoid being misled by misleading retrieved passages in RAG question answering. It improves F1 and exact match scores across five QA benchmarks compared to existing entropy-based selection rules.

EntropyMoE introduces an entropy-aware Mixture-of-Experts architecture for tokenizer-free LLMs, using dynamic byte patches as routing units to enable sparse conditional computation. Experiments show it achieves the lowest held-out bits-per-byte among baselines while maintaining downstream accuracy.

This paper proposes predicting task difficulty for LLM agents without running expensive rollouts, studying the problem across 17 agentic benchmarks and showing that token-level entropy is a useful predictive signal.

Analyzes how per-token LLM pricing creates incentives for verbose output, and proposes low-entropy prompt constraints (FAOA) to reduce cost and increase semantic density.

This paper investigates whether RLVR-trained LLMs branch out to discover heterogeneous inferences, using maze-solving experiments and BODHI-Trees to show that policy entropy collapse is accompanied by reduced semantic branching entropy, limiting rollout diversity.

This paper tests entropy-based pruning for chain-of-thought compression across models and tasks, finding it offers no advantage over random pruning, and that low-entropy token retention only helps on math benchmarks due to numeric tokens. It provides causal evidence that reasoning information is distributed across the full chain rather than concentrated in a few identifiable tokens.

An insightful explainer on the true meaning of entropy, contrasting the common 'disorder' metaphor with a probabilistic interpretation using dice rolling analogies.

A detailed walkthrough explains how Claude Shannon's 1948 information theory underlies LLMs and shows that the 'next-token prediction' story is misleading, linking compression and prediction mathematically.

This paper measures how much formal semantic structure explains human label variation in natural language inference (NLI) using ChaosNLI data, finding group-level effects on entropy but item-level ceilings and null composition effects.

This paper compares semantic search dynamics between humans and LLMs using verbal fluency data, finding that humans exhibit more variable and exploratory search patterns that current models fail to reproduce.

This study uses semantic entropy, an NLP embedding-based metric, to compare semantic memory navigation between blind and sighted individuals. Results show that visual experience influences entropy patterns, with sighted individuals having higher entropy for abstract concepts while blind individuals exhibit higher entropy for visually salient concrete concepts.