sparsity

Tag

Cards List
#sparsity

Sparse Inter-Layer Dependencies of Transformer FFN Neurons

arXiv cs.LG ↗ · 2026-07-15 Cached

This paper introduces a training-free attribution method to identify sparse inter-layer dependencies in Transformer FFN neurons, showing that small subsets of preceding activations suffice to preserve neuron activations with high fidelity.

0 favorites 0 likes
#sparsity

Mitigating Early Training Collapse in CTR Models

arXiv cs.LG ↗ · 2026-07-14 Cached

This paper analyzes the early training collapse phenomenon in deep neural models for click-through rate prediction and proposes mitigation strategies such as sparse feature removal and value filtering, demonstrating improvements on large-scale industrial datasets.

0 favorites 0 likes
#sparsity

Optimized Instance Alteration for Explaining and Assessing Robustness of Classifiers

arXiv cs.LG ↗ · 2026-07-09 Cached

This paper proposes a unified optimization framework to explain misclassifications and assess classifier robustness by sparse, interpretable instance alterations and a Tolerance Region Confusion Matrix.

0 favorites 0 likes
#sparsity

Entropy-Regularized Probabilistic Gates for Sparse Model Discovery in Scarce-Data Federated Learning

arXiv cs.LG ↗ · 2026-07-02 Cached

This paper proposes entropy-regularized probabilistic gates to maintain uncertainty in sparse federated optimization, improving sparsity recovery and test performance under data heterogeneity and scarce data.

0 favorites 0 likes
#sparsity

SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

arXiv cs.LG ↗ · 2026-06-26 Cached

SharQ introduces a training-free method combining activation sparsity and FP4 quantization for LLM inference, using sparse-dense decomposition and a unified FP4 weight payload. It achieves significant latency reduction and accuracy recovery over FP4-only baselines.

0 favorites 0 likes
#sparsity

Effects of sparsity and superposition on loss in simple autoencoders

arXiv cs.LG ↗ · 2026-06-18 Cached

This paper provides a mathematical analysis of superposition in neural networks, deriving upper and lower bounds on L2 reconstruction loss for simple autoencoders with power activation functions, corroborating empirical findings by Elhage et al.

0 favorites 0 likes
#sparsity

Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging

arXiv cs.LG ↗ · 2026-06-18 Cached

This paper investigates the 'sparsity curse' in merging RLVR (Reinforcement Learning with Verifiable Reward) models, finding that sparse updates cause near-orthogonal parameter directions that hinder aggregation, and proposes SAR-Merging, which uses Fisher information and sparsification to resolve conflicts and improve merging performance on math and coding tasks.

0 favorites 0 likes
#sparsity

ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation

arXiv cs.CL ↗ · 2026-06-17 Cached

ConSA is a framework that learns optimal assignment between full attention and sliding-window attention under a user-specified sparsity target, using L0 regularization and augmented Lagrangian constraint. It demonstrates consistent gains over rule-based baselines on LLMs at 0.6B and 1.7B scales.

0 favorites 0 likes
#sparsity

Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective

arXiv cs.CL ↗ · 2026-06-16 Cached

This paper proposes an inter-layer perturbation-absorption perspective for layer-wise sparsity in LLMs, showing that layers exhibit heterogeneous responses to pruning perturbations and introducing an absorption-aware correction that improves existing pruning methods by reducing perplexity and boosting accuracy.

0 favorites 0 likes
#sparsity

Simplex-Constrained Sparse Bagging: Transitioning from Uniform Priors to Sparse Posteriors in Ensemble Learning

arXiv cs.AI ↗ · 2026-06-15 Cached

Introduces Simplex-Constrained Sparse Bagging (SCSB), a post-training framework that optimizes estimator weights over the probability simplex using out-of-bag samples, achieving up to 96% ensemble compression and improved calibration.

0 favorites 0 likes
#sparsity

Beyond LoRA: Is Sparsity-Induced Adaptation Better?

arXiv cs.LG ↗ · 2026-06-15 Cached

This paper proposes sparsity-induced adaptations to LoRA, including Cheap LoRA (cLA) and a chained circulant variant (c³LA), and provides theoretical generalization bounds along with empirical evaluations showing up to 10% training time reduction and 15% peak GPU memory savings while maintaining competitive performance.

0 favorites 0 likes
#sparsity

Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

Hugging Face Daily Papers ↗ · 2026-06-11 Cached

This paper analyzes on-policy distillation (OPD), finding that OPD updates are sparse, distributed across layers and FFN-heavy, and retain geometric properties distinct from dense parameter rewriting. The sparse structure is operationally useful, but sparsity-inducing SGD underperforms AdamW due to heterogeneous gradient scales.

0 favorites 0 likes
#sparsity

Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs

arXiv cs.CL ↗ · 2026-06-10 Cached

This paper proposes a dense-to-sparse continual training method for LLMs, using a predictor-gated bank-wise sparsity to achieve 4x FFN sparsity, and demonstrates it on Qwen2.5-8B with long-context training.

0 favorites 0 likes
#sparsity

Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers

arXiv cs.CL ↗ · 2026-05-26 Cached

This paper introduces a grammatically-guided sparse attention mechanism for Transformers, aiming to improve efficiency and interpretability by leveraging linguistic structure.

0 favorites 0 likes
#sparsity

RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models

Hugging Face Daily Papers ↗ · 2026-05-26 Cached

RT-Lynx proposes using activation sparsity instead of weight sparsity to accelerate diffusion models, achieving up to 1.55× linear-layer speedup while maintaining generation quality, and is accepted at ICML 2026.

0 favorites 0 likes
#sparsity

Two-Valued Symmetric Circulant Matrices: Applications in Deep Learning

arXiv cs.LG ↗ · 2026-05-19 Cached

This paper proposes the Two-Valued Symmetric Circulant Matrix (TVSCM), a very sparse architecture that uses only two weights per layer to achieve over 80x parameter reduction on MNIST and MIT-BIH arrhythmia datasets while maintaining comparable accuracy, making it ideal for edge and tiny-ML platforms.

0 favorites 0 likes
#sparsity

$\phi$-Balancing for Mixture-of-Experts Training

arXiv cs.LG ↗ · 2026-05-18 Cached

This paper proposes φ-balancing, a principled framework for load balancing in Mixture-of-Experts models that directly targets population-level expert balance using convex duality and mirror descent, achieving more stable expert utilization and outperforming prior methods on reasoning and code generation benchmarks.

0 favorites 0 likes
#sparsity

Neural Activation Patterns Across Language Model Architectures: A Comprehensive Analysis of Cognitive Task Performance

arXiv cs.CL ↗ · 2026-05-18 Cached

This paper analyzes neural activation patterns across six LLM architectures on cognitive tasks, revealing differences in attention entropy and sparsity between encoder and decoder models.

0 favorites 0 likes
#sparsity

@hardmaru: The human brain is incredibly efficient because it only activates the specific neurons needed for a thought. Modern LLM…

X AI KOLs Timeline ↗ · 2026-05-08 Cached

This paper introduces TwELL and Hybrid sparse formats with custom CUDA kernels to efficiently leverage unstructured sparsity in LLMs, achieving over 20% faster training and inference on H100 GPUs while reducing energy and memory usage.

0 favorites 0 likes
← Previous
← Back to home

Submit Feedback