Tag
A detailed walkthrough explains how Claude Shannon's 1948 information theory underlies LLMs and shows that the 'next-token prediction' story is misleading, linking compression and prediction mathematically.
A practitioner recommends a free 33-minute lecture on cross-entropy that reframes language models as compression rather than next-word prediction, likening it to a Stanford ML PhD qualifier.
This paper develops conservation laws for diffusion models using generalized extrinsic information transfer (GEXIT) functions, showing that the cross-entropy can be characterized as an integral of local information-theoretic derivatives along the noise path, unifying likelihood characterization for discrete and continuous diffusion.
The paper investigates whether weight norm directly controls the grokking delay in neural networks or if its effect is mediated by logit scale and softmax saturation under cross-entropy loss. Experiments show that the delay is almost entirely explained by the effective logit scale, with weight norm contributing negligibly.
This paper shows that cross-entropy and supervised contrastive learning are both forms of prototype learning on the hypersphere and proposes normalized losses (NTCE and NONL) that achieve Neural Collapse by design, outperforming standard methods.
This paper derives a closed-form upper bound for admissible learning-rate steps in belief-space dynamics using KL divergence and Bregman geometry, focusing on cross-entropy classification.