sparsity

The paper introduces Library of Layers (LoL), a transformer architecture that dynamically reorders its layers at inference time, unifying parameter reuse and sparsity tradeoffs across dense, depth-recurrent, and MoE baselines while enabling interpretable routing across reasoning, multilingual, and safety tasks.

The paper shows that sparsity-inducing regularization causes functional modularity to emerge in multitask neural networks, identifying compact task-selective subnetworks ('functional tokens') that can be recombined across pathways without retraining.

该论文系统比较了 2:4 半结构化稀疏化与量化在 LLM 推理中的取舍，发现在仍有更低精度可用时，加入 2:4 稀疏化既更慢又更不准确；只有在达到硬件量化下限（如 FP8/NVFP4）后，2:4 稀疏化才展现出 1.3–1.7× 的线性层加速价值，且需通过重训练恢复质量。

论文提出 LADDER（Layer-Wise Allocation via Damage-Driven Evaluation and Refinement），一种针对大语言模型的非均匀后训练剪枝方法，通过逐层评估最小化每移除参数的性能损失，生成嵌套掩码轨迹，在 70% 以上高稀疏度下表现显著优于基线。

Part 1 of a 3-part series arguing that on-policy RL, distillation, and near-policy SFT updates are sparse, showing that because the score has zero mean under the sampler's distribution, the expected gradient is a covariance and the Fisher a variance, bounding any update regardless of reward.

提出一种免重训练的方法，将预训练 Transformer 转化为可弹性伸缩的模型，通过联合排序并软剪枝单个 MLP 神经元和注意力头，在多种稀疏度目标下连续自适应计算预算，在 27 个视觉与文本 Transformer 编码器上实现最高 60% 稀疏度下的平滑性能衰减。

The post argues that Qwen3.8-Flash-Next's ultra-sparse MoE design (large expert pool, few experts activated per token) yields 3x the decoding speed of GLM-5.3-Flash on the same RTX 4090 48G, since bandwidth reads scale with activated parameters. It frames extreme expert-granularity sparsity as a key architectural direction for local-model users ahead of Qwen4.

The paper presents FCx, an algorithm for generating feasible counterfactual explanations by using a modified Variational Autoencoder with causal inference to ensure modifications are realistic, low-cost, and compatible with real-world changes.

The paper proposes OBC-Prune, a calibration method for pruning large reasoning models that identifies causally important reasoning circuits to improve accuracy and reduce inference overhead on benchmarks like MATH500 and LiveCodeBench.

The paper introduces a unified Bayesian variational framework combining spike-and-slab sparsity and Gaussian mixture quantization for high compression rates in large neural networks with minimal accuracy loss.

The paper proposes TREVIS, a method that uses a Tree Transformer Variational Auto-Encoder to learn sparse decision trees by optimizing in a continuous latent space, achieving good predictive performance with improved structural sparsity.

This paper proposes a Sparse-Activation-ReLU (SAR) layer for low-latency, energy-efficient virtual sensing, achieving significant improvements in latency-error-energy metrics and reducing errors through synthetic knowledge distillation.

The paper introduces the 'Compression Trinity' framework, jointly applying sparsity, quantization, and low-rank approximations to compress Large Language Models for improved efficiency and performance.

This paper introduces difference-informed pruning methods (Wisp, Wisp+, Whisper) for large language models, showing that preserving output differences improves sparsification across Llama 2 and 3.1 models up to 405B parameters.

This paper studies identifiability in matrix tri-factorisation, showing that sparsity constraints can induce unique solutions for the factorisation problem.

This paper reports that deep reinforcement learning agents using frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations, compressing task-relevant information through very few neurons without any sparsity-inducing objective.

Proposes neuromorphic masked diffusion language models (N-MDLMs) that integrate block diffusion with spike-based neuromorphic computation to improve throughput and energy efficiency by leveraging sparsity and generating multiple tokens per parameter access, analyzed via a roofline-inspired model.

This paper introduces Neural Atom Prevalence (NAP), a Bayesian framework for structured node-level model selection that achieves high sparsity, accuracy, and uncertainty quantification in neural networks.

This paper introduces a compound sparsity framework for LLMs that combines static parameter pruning with dynamic token-level computation, showing that mixing both mechanisms outperforms single-dimension compression and delays performance degradation.

Discussion of LatentMoE architecture with extreme sparsity (16/896 experts) and Kimi Delta Attention, claiming 2.5x more efficient scaling, and speculation about Kimi K3 model capabilities.