Entropy as a Structural Prior: How a Log-Barrier on DiT Belief Space Drives Musical Diversity and Development
Summary
This paper introduces the Eisbach log-barrier, a parameter-free weight derived from the entropy of DiT output's spatial energy distribution, which when applied to LoRA fine-tuning of Stable Audio 3 improves musical diversity and thematic development without causing mode collapse.
View Cached Full Text
Cached at: 06/08/26, 11:15 AM
Paper page - Entropy as a Structural Prior: How a Log-Barrier on DiT Belief Space Drives Musical Diversity and Development
Source: https://huggingface.co/papers/2606.07207
Abstract
Confidence-based loss weighting via entropy-derived log-barrier enables improved audio generation through adaptive gradient scaling in supervised diffusion training.
Confidence-based loss weightingis usually avoided ingenerative modelsbecause it accelerates errors when the model is confidently wrong, but this intuition breaks down insupervised diffusion training. We introduce theEisbach log-barrier, a parameter-free weight derived from theentropyof theDiT output’sspatial energy distribution: highentropydamps the gradient, while lowentropypreserves it. Applied toLoRA fine-tuningofStable Audio 3Medium onMusicCaps, it unexpectedly yields stronger thematic development, clearer acoustic differentiation, and higher textural diversity than unweighted training, the opposite of mode collapse. This works because in supervised diffusion the gradient direction is locked to ground truth, so confidence only scales the step size, and becausetemporal entropydownweights flat samples while preserving high-contrast ones. The result is an online, self-referentialdata curriculumthat emerges purely from theforward pass, with analyzednoise-level dynamicsand testable predictions.
View arXiv pageView PDFProject pageAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.07207 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.07207 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.07207 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction
This preregistered reproduction study validates that the shape of chain-of-thought entropy trajectories predicts large language model answer correctness, while the total entropy drop is inconsistent across settings, and explores final-step entropy as an improved metric.
Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning
This paper introduces Conditional Entropy Shaping (CES), a framework that dynamically controls token-level response entropy in LLMs to balance reasoning depth and conciseness, achieving improved accuracy while reducing response length on mathematical benchmarks.
When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions
This paper investigates when chain-of-thought reasoning is beneficial for LLMs, showing that early-stage entropy dynamics reliably indicate reasoning utility, and introduces EDRM, a lightweight, training-free framework that adaptively selects inference strategies to achieve significant token savings while maintaining or improving accuracy.
Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMs
This paper proposes E³RL, a reinforcement learning method that uses dynamic epistemic entropy thresholds to enable LLMs to excise local logical defects during generation, overcoming the autoregressive curse in long-horizon reasoning and achieving state-of-the-art results on mathematical reasoning benchmarks like AIME.
Revisiting the Uniform Information Density Hypothesis in LLM Reasoning
This paper revisits the Uniform Information Density (UID) hypothesis in the context of LLM reasoning, introducing an entropy-based framework to quantify information flow uniformity. Across seven reasoning benchmarks, the authors find that high-quality reasoning exhibits local uniformity in step transitions but global non-uniformity in trajectory structure, suggesting LLM reasoning differs fundamentally from human communication patterns.