Tag
This paper applies Marchenko-Pastur random matrix theory to pre-trained attention weights, separating each projection matrix into a random-like bulk and spectral outliers. Causal experiments show zeroing these outliers in Mistral-7B drives performance near random chance, revealing that spectral outliers encode dominant learned structure across 11 transformers.
This paper develops a spectral framework for analyzing rotary phase alignment, semantic continuity, and representation drift in transformer language models, proposing a bounded spectral method and distinguishing internal coherence from execution-boundary governance.
The paper reveals that preference-based post-training induces parameter updates with a spectral head-tail organization, where a compact head carries the dominant behavioral shift and a weak tail is necessary for full solution recovery, recasting alignment as structured update reorganization rather than monolithic correction.
This paper identifies spectral drift in internal activations of neural networks during misclassifications and introduces Self-Detecting Neural Networks (SDNN) that monitors spectral dynamics to detect failures, achieving 79% AUROC on CIFAR-10, outperforming confidence-based methods by 25-30 percentage points.
Proposes Topologically Regularized Side-Path (TRSP) to mitigate representation collapse in LLMs by balancing spectral trade-offs between mixing efficiency and information capacity, achieving significant gains on long-context benchmarks.
This paper introduces a recurrent sinusoidal architecture for implicit neural representations that achieves higher fidelity with fewer parameters and optimization steps by exploiting harmonic line spectrum enrichment through sinusoidal recurrence.
This paper introduces SEB-Cal, a method that augments output-space calibration with spectral features (band energy, entropy, peak dominance, phase stability) to improve selective reliability estimation in time-series classification, achieving higher Corr-AUROC and lower [email protected] across multiple datasets.
This paper derives the eigenvalues of the Hessian for linear neural networks of arbitrary width and depth, showing that sharpness relates to maximum class proportion for classification tasks with MSE loss, and empirically validates the predictions.
This paper introduces SPARC, a spectral-algebraic theory explaining the self-correction blind spot in autoregressive language models, where models fail to correct their own errors but can fix identical external errors. The theory proves the blind spot arises when the spectral radius of an error-propagation operator is at least one, derives a threshold for correction markers, and provides convergence guarantees for RL-based self-correction training.
This paper analyzes spectral patterns in pretrained GPT-2-style language models and tests whether these patterns can be used for initialization, finding that coarse spectral matching does not improve pretraining performance over standard methods.
This paper investigates the spectral properties of the QK operator in attention heads, showing that the positional scheme (RoPE, learned absolute, ALiBi) sets a default spectral algebra that acts as a fingerprint consolidated after function rather than a hard constraint.
The paper introduces Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains the spectral core of RL updates to preserve reasoning gains, remove interference, and enable model merging across experts, achieving strong performance with minimal parameters.
This paper studies the geometric properties of chain-of-thought trajectories in the hidden state space of transformers, introducing effective dimension and kinematic features to predict task hardness and solution correctness from early tokens.
This paper presents NightVision, an attack that uses restrictive black-box API access to estimate hidden dimension, depth, and parameter count of large language models. It exploits a novel common-set prompting technique and spectral analysis, achieving high accuracy on open-source models.
This paper introduces Multi-Adapter PPO, a reinforcement learning framework with cross-attention for wavelength selection in LIBS quantitative analysis, achieving 28.4% better composite scores and 45.2% improvement in prediction accuracy over traditional methods on steel and coal datasets.
An explanation of why diffusion models work well for images: low-frequency spectral components dominate, so denoising recovers coarse structure first, then fine detail — analogous to spectral autoregression.
This paper introduces the Multi-Scale Feature Attention Network (MSFAN), a deep learning architecture for classifying 12 types of polymers using THz Dual-Comb Spectroscopy, achieving 85.2% accuracy and outperforming state-of-the-art models.
Resonate is a low-latency, low-memory algorithm for perceptually relevant spectral analysis of audio signals, using resonator models with exponentially weighted moving averages.
This paper presents an exact decomposition of the curvature exponent α in neural network loss landscapes, explaining why it varies across layer types. It introduces the spectral alignment decomposition and derives a spectral transfer identity linking curvature, gradient rank decay, and Hessian exponents, validated across architectures and datasets.
BitsMoE introduces a spectral-energy-guided bit allocation framework for quantizing Mixture-of-Experts LLMs, achieving substantial accuracy improvements and speedups under ultra-low-bit quantization.