Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

Hugging Face Daily Papers Papers
neural-spectral-capacity architecture-optimization transformer pruning training-free dynamic-programming

Summary

The paper introduces Neural Spectral Capacity (NSC), a training-free metric based on the singular-value spectrum to evaluate and optimize neural network architectures, with a dynamic programming method for optimal design under constraints.

Modern Transformer design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure: two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently. We propose Neural Spectral Capacity (NSC), a closed-form scalar grounded in the singular-value spectrum of each weight matrix. Under standard random initialization, the Marchenko-Pastur law renders NSC computable from the architectural specification alone, with no model instantiation, data, or gradients. Its layer-wise additive structure admits NSC-DP, an exact dynamic-programming solver returning the architecture globally maximizing NSC under resource constraints in seconds on a CPU -- a guarantee that black-box search over existing training-free proxies cannot provide. Empirically, NSC outperforms #Params, #FLOPs, and representative training-free proxies in ranking across seven Transformer and CNN families (on FlexiBERT, τ= 0.505 on pairs differing in #Params by less than 10%, where #Params collapses to 0.082); NSC-DP discovers a Transformer-XL architecture on WikiText-103 that beats the human-designed baseline in 2 seconds; and prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks without any calibration data, about 5900x faster than the strongest training-free proxy baseline.
Original Article
View Cached Full Text

Cached at: 09/25/26, 07:44 AM

Paper page - Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

Source: https://huggingface.co/papers/2609.23087

Abstract

ModernTransformerdesignandcompressionbothreducetoallocatingcapacityunderabudget.Thestandardscalarsforthesedecisions,#Paramsand#FLOPs,capturesizeandcomputebutnotarchitecturalstructure:twoarchitectureswithidenticalparameterbudgetsbutdifferentdepth-width,head,orFFNallocationsreceiveidenticalscoresyetbehavedifferently.WeproposeNeuralSpectralCapacity(NSC),aclosed-formscalargroundedinthesingular-valuespectrumofeachweightmatrix.Understandardrandominitialization,theMarchenko-PasturlawrendersNSCcomputablefromthearchitecturalspecificationalone,withnomodelinstantiation,data,orgradients.Itslayer-wiseadditivestructureadmitsNSC-DP,anexactdynamic-programmingsolverreturningthearchitecturegloballymaximizingNSCunderresourceconstraintsinsecondsonaCPU--aguaranteethatblack-boxsearchoverexistingtraining-freeproxiescannotprovide.Empirically,NSCoutperforms#Params,#FLOPs,andrepresentativetraining-freeproxiesinrankingacrosssevenTransformerandCNNfamilies(onFlexiBERT,τ=0.505onpairsdifferingin#Paramsbylessthan10%,where#Paramscollapsesto0.082);NSC-DPdiscoversaTransformer-XLarchitectureonWikiText-103thatbeatsthehuman-designedbaselinein2seconds;andprunesLLaMA-7Btothebest5.7Bmodelacrosseightcommonsensereasoningtaskswithoutanycalibrationdata,about5900xfasterthanthestrongesttraining-freeproxybaseline.

View arXiv pageView PDFGitHub2Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.23087 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.23087 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.23087 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Anytime Training with Schedule-Free Spectral Optimization

arXiv cs.LG

This paper introduces SF-NorMuon, a schedule-free spectral optimizer that matches or exceeds tuned AdamW on language models up to 772M parameters, with theoretical guarantees for stationarity and long-horizon stability.