Tag
Details a method to run a 13 million parameter ASR Conformer model directly on a microcontroller, highlighting advances in edge AI deployment.
Discusses methods or research on reducing the size of model parameters, likely focusing on techniques like pruning or quantization to improve efficiency.
KDFlow is a novel knowledge distillation framework for large language models that uses a decoupled architecture with SGLang for teacher inference and FSDP2 for student training, achieving 1.44x to 6.36x speedup over existing frameworks.
Introduces qZACH-ViT, a quantization-aware extension of ZACH-ViT with recursive intrinsic explanations, and Recursive Attribution-Stabilized Optimization (RASO) for stable attribution gradients. Achieves high prediction agreement and speedups on MedMNIST datasets after INT8 conversion.
PrismML announces Bonsai 27B, a binary-quantized version of Qwen3.6 27B that runs on a phone using only 1.125 bits per weight, claiming 89.5% intelligence retention. The model is being independently tested by @sudoingX to verify performance.
The author claims that ternary decomposition performs as well as Q4_K_M quantization while using slightly more VRAM, and is fully ternary and PTQ without QAT.
Announcement of Hy3 1-bit quantized model with 89-93 GB memory footprint.
Prism ML is in talks with Apple to deploy model-shrinking technology that would allow powerful AI models to run directly on iPhones, improving privacy and reducing reliance on cloud inference.
EvoLP is a self-evolving latency predictor for neural network models on edge devices, designed to guide model compression while satisfying strict latency constraints. It outperforms prior methods across multiple edge devices and model variants.
This paper proposes an efficient GPU inference method for LLMs with moderate unstructured sparsity, introducing a three-layer matrix storage format and a SpMM kernel that jointly utilizes sparse tensor cores and CUDA cores, achieving up to 1.64× kernel-level speedup over SpInfer and up to 1.41× end-to-end speedup over FlashLLM.
This paper proposes signed symmetric quantization for few-bit integers, which assigns the extra representable value to the dominant outlier tail, improving quantization error and perplexity in LLMs with no extra inference cost compared to standard symmetric quantization.
This paper proposes a unified approach to interpret knowledge distillation in LLMs using game-theoretic interactions, discovering that distillation sparsifies interactions, and introduces a loss function CIP to improve performance.
This paper proposes a structured pruning method for LLMs that addresses distribution mismatch, sign-information loss, and outlier influence when adapting unstructured pruning techniques, achieving comparable accuracy with 1.56-1.57x speedup on models like Llama-3-8B and Vicuna-v1.5-13B.
This tweet explains how to use dlt to collect traces and replace a 744B parameter agent with a much smaller 1.7B model in four steps.
This paper investigates knowledge distillation for time series classification using FCN, Inception, and ConvTran architectures on the UCR Archive, showing significant parameter reduction and performance maintenance.
Explains three uses of distillation in frontier model post-training pipelines: compressing large models into small ones, merging RL experts, and self-teaching. Includes a write-up detailing which models use each method.
This paper proposes a controllability–observability framework for compressing deep neural networks by reducing hidden-state redundancy, demonstrating significant compression with minimal accuracy loss on MNIST and CIFAR-10.
NVIDIA's paper introduces Puzzle-75B-A9B, a compressed hybrid MoE model that doubles server throughput while preserving quality, enabling cost-effective deployment of large language models.
NVIDIA releases Nemotron-Labs-3-Puzzle-75B-A9B, a compressed hybrid MoE LLM derived from Nemotron-3-Super, achieving approximately 2× higher server throughput and improved concurrency while maintaining strong accuracy across reasoning, coding, and long-context benchmarks.
This paper proposes Threshold Gating (TG) as a unified primitive for neural nonlinearity, showing that standard activation functions can be expressed as instances of TG. The authors validate their approach by converting pretrained models across various architectures without retraining and discuss hardware benefits.