Entropy as a Structural Prior: How a Log-Barrier on DiT Belief Space Drives Musical Diversity and Development
Summary
This paper introduces the Eisbach log-barrier, a parameter-free weight derived from the entropy of DiT output's spatial energy distribution, which when applied to LoRA fine-tuning of Stable Audio 3 improves musical diversity and thematic development without causing mode collapse.
View Cached Full Text
Cached at: 06/08/26, 11:15 AM
Paper page - Entropy as a Structural Prior: How a Log-Barrier on DiT Belief Space Drives Musical Diversity and Development
Source: https://huggingface.co/papers/2606.07207
Abstract
Confidence-based loss weighting via entropy-derived log-barrier enables improved audio generation through adaptive gradient scaling in supervised diffusion training.
Confidence-based loss weightingis usually avoided ingenerative modelsbecause it accelerates errors when the model is confidently wrong, but this intuition breaks down insupervised diffusion training. We introduce theEisbach log-barrier, a parameter-free weight derived from theentropyof theDiT output’sspatial energy distribution: highentropydamps the gradient, while lowentropypreserves it. Applied toLoRA fine-tuningofStable Audio 3Medium onMusicCaps, it unexpectedly yields stronger thematic development, clearer acoustic differentiation, and higher textural diversity than unweighted training, the opposite of mode collapse. This works because in supervised diffusion the gradient direction is locked to ground truth, so confidence only scales the step size, and becausetemporal entropydownweights flat samples while preserving high-contrast ones. The result is an online, self-referentialdata curriculumthat emerges purely from theforward pass, with analyzednoise-level dynamicsand testable predictions.
View arXiv pageView PDFProject pageAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.07207 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.07207 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.07207 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning
This paper introduces Conditional Entropy Shaping (CES), a framework that dynamically controls token-level response entropy in LLMs to balance reasoning depth and conciseness, achieving improved accuracy while reducing response length on mathematical benchmarks.
When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions
This paper investigates when chain-of-thought reasoning is beneficial for LLMs, showing that early-stage entropy dynamics reliably indicate reasoning utility, and introduces EDRM, a lightweight, training-free framework that adaptively selects inference strategies to achieve significant token savings while maintaining or improving accuracy.
Revisiting the Uniform Information Density Hypothesis in LLM Reasoning
This paper revisits the Uniform Information Density (UID) hypothesis in the context of LLM reasoning, introducing an entropy-based framework to quantify information flow uniformity. Across seven reasoning benchmarks, the authors find that high-quality reasoning exhibits local uniformity in step transitions but global non-uniformity in trajectory structure, suggesting LLM reasoning differs fundamentally from human communication patterns.
Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
This paper proposes Adaptive Entropy Regularization (AER), a framework that dynamically balances exploration and exploitation in LLM reinforcement learning by addressing policy entropy collapse through difficulty-aware coefficient allocation and initial-anchored target entropy. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements in both accuracy and exploration capability.
Human-Centered Learning Mechanics: A Dynamical Framework for Entropy-Regulated Representation Learning
This paper proposes Human-Centered Learning Mechanics (HCLM), a dynamical and information-theoretic framework for studying open and controlled learning systems. It formalizes entropy regularization through effective information force, derives convergence and generalization results, and provides a conditional interpretation of scaling-law behavior.