Tag
This paper proposes a structured pruning method for LLMs that addresses distribution mismatch, sign-information loss, and outlier influence when adapting unstructured pruning techniques, achieving comparable accuracy with 1.56-1.57x speedup on models like Llama-3-8B and Vicuna-v1.5-13B.
This tweet explains how to use dlt to collect traces and replace a 744B parameter agent with a much smaller 1.7B model in four steps.
This paper investigates knowledge distillation for time series classification using FCN, Inception, and ConvTran architectures on the UCR Archive, showing significant parameter reduction and performance maintenance.
Explains three uses of distillation in frontier model post-training pipelines: compressing large models into small ones, merging RL experts, and self-teaching. Includes a write-up detailing which models use each method.
This paper proposes a controllability–observability framework for compressing deep neural networks by reducing hidden-state redundancy, demonstrating significant compression with minimal accuracy loss on MNIST and CIFAR-10.
NVIDIA's paper introduces Puzzle-75B-A9B, a compressed hybrid MoE model that doubles server throughput while preserving quality, enabling cost-effective deployment of large language models.
NVIDIA releases Nemotron-Labs-3-Puzzle-75B-A9B, a compressed hybrid MoE LLM derived from Nemotron-3-Super, achieving approximately 2× higher server throughput and improved concurrency while maintaining strong accuracy across reasoning, coding, and long-context benchmarks.
This paper proposes Threshold Gating (TG) as a unified primitive for neural nonlinearity, showing that standard activation functions can be expressed as instances of TG. The authors validate their approach by converting pretrained models across various architectures without retraining and discuss hardware benefits.
Introduces Variable Bit-width Quantization (VBQ), a training-time method where each group of 64 weights learns its own bit-width (1,2,4,8) via Gumbel-Softmax relaxation. VBQ discovers a heterogeneous allocation that yields a 'bigger-but-smaller' regime, e.g., a 131M parameter model at 1.82 mean bits beats a 55M FP16 model while using less storage, and a 1.46B model matches a 593M FP16 with ~3.7x less storage.
PuzzleMoE introduces a pairwise dual-mask expert merging algorithm and bit-level packing technique to compress large Mixture-of-Experts models, maintaining performance while reducing storage and accelerating inference.
Proposes Generic TB-Coverage, a coverage-aware expert pruning method for sparse Mixture-of-Experts language models that uses only generic text corpora for calibration and preserves cross-corpus expert coverage, improving accuracy and reducing perplexity degradation.
This paper investigates the effects of domain-specific expert pruning on both utility and factual reliability of Mixture-of-Experts (MoE) models in the biomedical domain. It finds that moderate pruning preserves in-domain utility without immediate reliability loss, but extreme pruning increases hallucination risks, and generalization degrades rapidly in cross-domain settings.
NVIDIA's 4-bit quantized Qwen3.6-27B (NVFP4) is found to be near-lossless compared to the full bf16 model, with behavioral differences being minor and random rather than systematic, making it a practical drop-in replacement.
A developer is rebuilding Gemma 4 31b into a smaller 26b model by removing weak SWA layers, adding attention-based residual networks (from Moonshot), and using topK logit targets for retraining, aiming for better long context and performance.
Introduces Cross-Space Distillation, a method to transfer knowledge from modern high-capacity diffusion models to compact student models with different latent spaces using a lightweight latent interface called Bridge, enabling quality improvements without modifying the student backbone.
A developer implemented a --skip-layers flag for a llama.cpp fork, allowing users to skip entire transformer blocks at load time with minimal performance impact, based on recent papers. This provides another method for fitting large models on limited hardware, complementing quantization and bake-time pruning.
FlexMoE proposes a one-for-all nested intra-expert pruning method for MoE language models, enabling multiple deployable subnetworks from a single training run with minimal performance loss.
Introduces Proxy-KD, a novel method for distilling knowledge from black-box large language models (like GPT-4) into smaller models using a proxy model, surpassing both traditional black-box and white-box KD techniques.
Clark Labs released Clark Air Sana 1.6B, a ternary-quantized version of the Sana 1.6B text-to-image transformer that is 8.6× smaller than FP16 while maintaining near-FP16 quality, enabling efficient deployment.
A calibration-aware Q4_K_M quantization of Qwen3.5 0.8B using SpectralQuant recovers 96.5% of the BF16 performance gap compared to the standard llama.cpp Q4_K_M quant.