theory

The article questions whether theoretical principles still guide machine learning practices, highlighting how many once-standard theories have been challenged by empirical evidence.

A survey paper examining the expressive power of transformers as language recognizers, using concepts and methods from circuit complexity to compare them with classical models of computation.

This paper introduces low interaction rank as a unified theoretical framework for multiplicative dual-encoder networks, covering approximation, sample complexity, normalization, and identifiability, with experiments on operator learning and CLIP models.

This paper proves that a single normalized nonnegative kernel-attention head requires exponentially many features to solve a simple Min-IP task on three-token sequences, whereas dense softmax attention solves it with constant temperature and m-dimensional scores, highlighting a fundamental expressive-power gap between kernel and full attention.

This paper reviews the concept of 'perspective' in NLP, proposes a hierarchy of perspective-related concepts along a specificity axis, and demonstrates how this hierarchy can help researchers choose appropriate operationalizations.

A theory paper introducing Decoupled Descent (DD), a training method that uses approximate message passing Onsager corrections to enforce asymptotic equality between training and test error during gradient descent, potentially enabling better stopping and hyperparameter tuning.

This paper studies the joint effect of memory width and batch depth in stochastic Lipschitz bandits, characterizing the minimax pseudo-regret tradeoff up to logarithmic factors and showing that state width and update depth are not interchangeable.

This paper studies the sample complexity of policy learning under the mu-resets interaction protocol in reinforcement learning, resolving a question about the role of policy realizability and showing horizon dependence is exponential under all-policy concentrability and sqrt-exponential under pushforward concentrability.

This paper introduces a constant-aware comparison protocol for average-reward reinforcement learning regret bounds, deriving an explicit finite lower certificate for communicating MDPs and improving published coefficients.

PRISM introduces a theory for designing reference processes in Schrödinger bridge models, showing that under finite computational budgets the optimal reference noise spectrum is determined by the sensor's information destruction spectrum. Experiments confirm the theory in Gaussian settings and identify where real images deviate.

This paper proves that across a broad class of ANNs, inference logic can be reformulated as sparse symbolic interactions, supported by mathematical criteria and extensive experiments, offering novel insights into explainability and generalization.

This paper presents a multiscale reward hedging method for learning from correct demonstrations, extending guarantees to continuous reward classes with a horizon-free bound via metric entropy, and shows polynomial-time cases for specific settings.

This paper analyzes how the parameterization of query-key and output-value circuits in self-attention models affects attention sharpness during training. Through gradient-flow analysis, they show that faster query-key learning relative to output-value learning produces sharper attention, improving interpretability without sacrificing predictive performance.

This paper introduces a simple Dirichlet-based forecaster that achieves optimal simultaneous multiclass U-calibration rates, closing the known dimension gap in regret bounds for bounded proper losses and removing extra additive terms for smooth losses.

The author describes spending seven days straight using the AI models GPT 5.6 Sol and Fable 5 to solve a 25-year-old open problem in wireless communication theory, noting that verification was the biggest bottleneck.

The paper proposes a mean-field framework to model chain-of-thought reasoning in LLMs as a guided discovery process on a clue graph, deriving an ODE for the fraction of discovered clues and validating it experimentally.

This paper introduces Relational Response Fields (RRF), a theoretical framework for determining when black-box LLM responses can be reliably recovered under corruption, establishing identifiability conditions and minimax bounds that separate response consistency from truth.

This paper develops an ℓ0-type stability theory for subdominant (minmax) ultrametrics, proving that sparse edits propagate only through the minimum spanning tree and deriving Hamming–Lipschitz bounds on changed ultrametric entries. Experiments on deep-embedding graphs and clustering tasks demonstrate the utility of the resulting structural scores as vulnerability diagnostics.

This paper provides theoretical guarantees for distinguishing LLM responses by modeling token embeddings as trajectories of a dynamical system, proving exponential decay of misclassification probability and characterizing cross-embedding generalization.

This theoretical paper proposes a driven-nucleation rate law to explain capability emergence, plasticity loss, and circuit control in language models, supported by experiments on Pythia and a controlled gated-attention model.