sparsity

The paper presents FCx, an algorithm for generating feasible counterfactual explanations by using a modified Variational Autoencoder with causal inference to ensure modifications are realistic, low-cost, and compatible with real-world changes.

The paper proposes OBC-Prune, a calibration method for pruning large reasoning models that identifies causally important reasoning circuits to improve accuracy and reduce inference overhead on benchmarks like MATH500 and LiveCodeBench.

The paper introduces a unified Bayesian variational framework combining spike-and-slab sparsity and Gaussian mixture quantization for high compression rates in large neural networks with minimal accuracy loss.

The paper proposes TREVIS, a method that uses a Tree Transformer Variational Auto-Encoder to learn sparse decision trees by optimizing in a continuous latent space, achieving good predictive performance with improved structural sparsity.

This paper proposes a Sparse-Activation-ReLU (SAR) layer for low-latency, energy-efficient virtual sensing, achieving significant improvements in latency-error-energy metrics and reducing errors through synthetic knowledge distillation.

The paper introduces the 'Compression Trinity' framework, jointly applying sparsity, quantization, and low-rank approximations to compress Large Language Models for improved efficiency and performance.

This paper introduces difference-informed pruning methods (Wisp, Wisp+, Whisper) for large language models, showing that preserving output differences improves sparsification across Llama 2 and 3.1 models up to 405B parameters.

This paper studies identifiability in matrix tri-factorisation, showing that sparsity constraints can induce unique solutions for the factorisation problem.

This paper reports that deep reinforcement learning agents using frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations, compressing task-relevant information through very few neurons without any sparsity-inducing objective.

Proposes neuromorphic masked diffusion language models (N-MDLMs) that integrate block diffusion with spike-based neuromorphic computation to improve throughput and energy efficiency by leveraging sparsity and generating multiple tokens per parameter access, analyzed via a roofline-inspired model.

This paper introduces Neural Atom Prevalence (NAP), a Bayesian framework for structured node-level model selection that achieves high sparsity, accuracy, and uncertainty quantification in neural networks.

This paper introduces a compound sparsity framework for LLMs that combines static parameter pruning with dynamic token-level computation, showing that mixing both mechanisms outperforms single-dimension compression and delays performance degradation.

Discussion of LatentMoE architecture with extreme sparsity (16/896 experts) and Kimi Delta Attention, claiming 2.5x more efficient scaling, and speculation about Kimi K3 model capabilities.

This paper introduces a training-free attribution method to identify sparse inter-layer dependencies in Transformer FFN neurons, showing that small subsets of preceding activations suffice to preserve neuron activations with high fidelity.

This paper analyzes the early training collapse phenomenon in deep neural models for click-through rate prediction and proposes mitigation strategies such as sparse feature removal and value filtering, demonstrating improvements on large-scale industrial datasets.

This paper proposes a unified optimization framework to explain misclassifications and assess classifier robustness by sparse, interpretable instance alterations and a Tolerance Region Confusion Matrix.

This paper proposes entropy-regularized probabilistic gates to maintain uncertainty in sparse federated optimization, improving sparsity recovery and test performance under data heterogeneity and scarce data.

SharQ introduces a training-free method combining activation sparsity and FP4 quantization for LLM inference, using sparse-dense decomposition and a unified FP4 weight payload. It achieves significant latency reduction and accuracy recovery over FP4-only baselines.

This paper provides a mathematical analysis of superposition in neural networks, deriving upper and lower bounds on L2 reconstruction loss for simple autoencoders with power activation functions, corroborating empirical findings by Elhage et al.

This paper investigates the 'sparsity curse' in merging RLVR (Reinforcement Learning with Verifiable Reward) models, finding that sparse updates cause near-orthogonal parameter directions that hinder aggregation, and proposes SAR-Merging, which uses Fisher information and sparsification to resolve conflicts and improve merging performance on math and coding tasks.