GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

Hugging Face Daily Papers Papers
transformer-compression cross-layer-factorization training-free geometry-preservation model-compression deep-learning

Summary

The paper introduces GeoPair, a training-free framework for transformer compression that optimizes cross-layer factorizations while preserving activation geometries, achieving state-of-the-art results across diverse architectures.

Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on heuristic grouping strategies that disregard layer-specific activation geometries. We introduce a principled, training-free framework that sequentially optimizes cross-layer weight pairings and shared-dictionary factorizations. Rather than forcing weights of adjacent layers to share a basis or heuristically merging activation statistics, our approach identifies structurally compatible projections and learns a shared representation that better preserves each layer's distinct calibration geometry. Coupled with structured sparsity, this yields highly efficient weight decompositions without sacrificing functional fidelity. Across diverse architectures, scales, and modalities, our method achieves state-of-the-art results, consistently outperforming independent structured weight decompositions and alternative pairwise weight factorizations, which operate under heuristic grouping strategies. By replacing heuristic engineering strategies with a convergent, optimization-driven pipeline, we establish a theoretically grounded foundation for scalable, transformer compression across different modalities.
Original Article
View Cached Full Text

Cached at: 09/24/26, 11:39 AM

Paper page - GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

Source: https://huggingface.co/papers/2609.25963

Abstract

Transformerarchitecturesexhibitcross-layerredundancies,yetpost-trainingcompressionpipelinestypicallyoptimizelayersinisolationorrelyonheuristicgroupingstrategiesthatdisregardlayer-specificactivationgeometries.Weintroduceaprincipled,training-freeframeworkthatsequentiallyoptimizescross-layerweightpairingsandshared-dictionaryfactorizations.Ratherthanforcingweightsofadjacentlayerstoshareabasisorheuristicallymergingactivationstatistics,ourapproachidentifiesstructurallycompatibleprojectionsandlearnsasharedrepresentationthatbetterpreserveseachlayer’sdistinctcalibrationgeometry.Coupledwithstructuredsparsity,thisyieldshighlyefficientweightdecompositionswithoutsacrificingfunctionalfidelity.Acrossdiversearchitectures,scales,andmodalities,ourmethodachievesstate-of-the-artresults,consistentlyoutperformingindependentstructuredweightdecompositionsandalternativepairwiseweightfactorizations,whichoperateunderheuristicgroupingstrategies.Byreplacingheuristicengineeringstrategieswithaconvergent,optimization-drivenpipeline,weestablishatheoreticallygroundedfoundationforscalable,transformercompressionacrossdifferentmodalities.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.25963

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.25963 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.25963 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.25963 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Fine-Tuning of Transformer models with Frames

arXiv cs.AI

The paper proposes FrameFT, a parameter-efficient fine-tuning method using sparse coefficients in a Fusion Frame basis, reducing memory footprint while achieving performance on par with or exceeding state-of-the-art PEFT techniques.