shannon-capacity

The paper proposes a Shannon Scaling Law that models LLM training as information transmission over a noisy channel, explaining non-monotonic performance phenomena like catastrophic overtraining and quantization-induced degradation, and demonstrating superior predictive accuracy over traditional scaling laws.