LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

Hugging Face Daily Papers Papers
llm scaling-laws shannon-capacity noisy-channel information-theory performance-degradation

Summary

The paper proposes a Shannon Scaling Law that models LLM training as information transmission over a noisy channel, explaining non-monotonic performance phenomena like catastrophic overtraining and quantization-induced degradation, and demonstrating superior predictive accuracy over traditional scaling laws.

Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, inducing a transition from monotonic improvement to U-shaped performance degradation. We validate our theory through experiments on Pythia and OLMo2 under perturbations, including Gaussian noise, quantization and supervised fine-tuning on math, QA and code tasks. The Shannon Scaling Law consistently outperforms classical scaling laws and recent perturbation-aware laws, achieving strong R^2 scores and accurately capturing loss basins missed by prior approaches. It also extrapolates: fitted on leq6.9B Pythia models with leq180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled R^2{=}0.847, while monotonic baselines collapse.
Original Article
View Cached Full Text

Cached at: 05/25/26, 02:35 AM

Paper page - LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

Source: https://huggingface.co/papers/2605.23901

Abstract

The Shannon Scaling Law models LLM training as information transmission over a noisy channel, explaining non-monotonic performance phenomena through signal-to-noise ratio interactions and demonstrating superior predictive accuracy over traditional scaling laws.

Existingscaling lawsfor Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such ascatastrophic overtrainingandquantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that modelsLLM trainingasinformation transmissionover anoisy channel, grounded in theShannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficientsignal-to-noise ratio(SNR) inevitably amplifies noise, inducing a transition from monotonic improvement to U-shaped performance degradation. We validate our theory through experiments onPythiaandOLMo2under perturbations, includingGaussian noise, quantization andsupervised fine-tuningon math, QA and code tasks. The Shannon Scaling Law consistently outperforms classicalscaling lawsand recent perturbation-aware laws, achieving strong R^2 scores and accurately capturingloss basinsmissed by prior approaches. It also extrapolates: fitted on leq6.9BPythiamodels with leq180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled R^2{=}0.847, while monotonic baselines collapse.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.23901

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.23901 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.23901 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.23901 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

A Theory of Training Profit-Optimal LLMs

arXiv cs.LG

This paper develops an economic model combining scaling laws with microeconomic theory to analyze profit-optimal training of large language models, considering trade-offs between model quality, training costs, and hardware efficiency.

Scaling Closed-Loop Feature Channel Configuration with LLMs

arXiv cs.LG

This paper scales a closed-loop LLM-based channel configuration search to 250 candidates per cycle, showing positive accuracy trends and improved parameter efficiency on CIFAR-100, and revealing architectural regularities in LLM-generated channel priors.

The Scaling Laws of Skills in LLM Agent Systems

arXiv cs.CL

This paper identifies two coupled scaling laws for skill libraries in LLM agent systems: routing accuracy decays logarithmically with library size, and execution dynamics show a rescue effect. The laws are validated across 15 models and over a million decisions, and law-guided optimization significantly improves performance.

The Asymmetric Harms of LLM Compression

arXiv cs.CL

This paper systematically evaluates LLM compression methods and reveals asymmetric harms, such as disproportionate loss of head knowledge and increased confidence in incorrect answers, which are masked by standard aggregate metrics.