Tag
This paper develops a theoretical framework for in-context learning on partial orders, analyzing identifiability, teaching cost, and representation limits with exact completion trichotomies.
A researcher presents a theoretical bound showing that similarity-based AI-text detectors (watermarking, retrieval) face a false-positive rate floor determined by the collision entropy of the text distribution, and asks for community feedback on the proof and connections to prior work.
This paper analyzes when random low-dimensional reparameterizations can train neural networks, deriving an orientation-resolved master formula for the random-slice residual and introducing RaMaN, a scalable framework that predicts required latent dimensions while dramatically reducing memory costs.
This paper introduces Power Law Graph Attention (PLGA) and the PLDR-LLM architecture, an exact generalization of scaled dot-product attention using input-generated bilinear operators. It presents theoretical results including an inference-collapse theorem, empirical stability measurements, and machine-checked proofs in Lean 4.
This paper theoretically analyzes support selection in continuous DAG learning, showing that smooth acyclicity constraints alone cannot rank supports beyond feasibility and deriving selection times for NOTEARS/DAGMA, with empirical audits on 320 trajectories.
This paper introduces Matrix Zonotopic Attention (MZAttn), a context-adaptive value projection for set transformers, and provides a theoretical analysis showing it can represent target operators that standard attention requires depth to approximate. Experiments demonstrate performance gains on high-rank combinatorial set-prediction tasks.
This paper presents a theoretical and empirical analysis showing that SFT suffers from task conflicts in multi-task LLM training while RL enables stable coexistence, proposing the Parallel-RL paradigm for efficient multi-task training.
This paper establishes the first convergence guarantees for the plain vector-form Adam optimizer under heavy-tailed stochastic noise, showing it converges to stationary points but with suboptimal iteration complexity, and improves to optimal rates with a known domain radius.
This paper introduces the Entropic Bound, a spectral measure of task-intrinsic capacity for transformers, proving that the intrinsic rank of the token-mixing operator provides a tight lower bound on required model capacity. It shows that while a naive transfer from linear attention fails for real attention, an attention-native intrinsic rank restores the full theoretical structure.
This paper provides the first finite-time convergence guarantees for the Natural Policy Gradient algorithm in finite-horizon Markov Decision Processes, proving sublinear and linear convergence rates under different step size regimes.
This paper establishes a rigorous quantitative connection between neural network score function approximation and the resulting distribution approximation in score-based diffusion models, proving that accurate score approximation leads to close distribution approximation in KL divergence, with an explicit bound.
This paper provides the first mathematical proof that low-rank decomposition and quantization are non-orthogonal when combined for LLM compression, leading to performance degradation, and proposes a novel Diagonal Adhesive Method (DAM) to mitigate this loss.
This paper studies whether alignment and bounded safety filters can fully eliminate harmful outputs from large language models, providing theoretical arguments and empirical evidence that harmful output rates plateau above zero under these constraints.
This paper provably shows that whether learning-rate cooldown helps in WSD schedules depends on the structure of gradient noise and whether the optimizer normalizes its update, explaining why cooldown can be ineffective for SGD but necessary for normalized methods.
This paper proposes correlation-aware contextual bandit algorithms that leverage surrogate reward signals from machine learning models for LLM routing, achieving improved accuracy-cost trade-offs and sample efficiency compared to standard baselines.
This paper introduces analytically tractable Bayesian information restricted diffusion (BIRD) models to study the memorization-generalization phase transition in diffusion models, finding that generation proceeds near the edge of memorization and that information restriction helps circumvent the curse of dimensionality.
This paper demonstrates that optimal learning rate scaling in deep scalar linear networks is inherently data-dependent, contradicting prior data-agnostic scaling rules. It shows that with data-dependent scaling, convergence becomes depth-independent, including at infinite depth.
This paper argues that the bulk of near-zero Hessian eigenvalues in neural networks arises from weakly broken symmetries of the network parametrization, showing that high-curvature directions are orthogonal to the symmetry subspace while the bulk lies within it.
This paper develops a sampling-complexity theory for in-context search in LLMs, showing that when reflections localize early mistakes, in-context search yields exponential improvements over base models, and validates predictions on real reasoning models.
This paper theoretically characterizes the minimax risk of KV cache compression in transformers, providing design principles for accurate compression under causal masking, and instantiates them in a practical algorithm with promising results on LongBench.