Tag
MiniCPM5-2B is a 2B-parameter open-source language model that achieves high intelligence density for edge deployment and ranks highly in several AI performance benchmarks.
The paper presents NanoSleep, a parameter-efficient hybrid temporal convolutional network for automatic sleep stage classification using single-channel EEG, designed for wearable and home-based monitoring on resource-constrained devices.
This paper presents Wiola, a 13M parameter decoder-only language model with novel components like Spiral Rotary Positional Encoding and Gated Spiral Attention to enhance parameter efficiency for small-scale, on-device language models.
This paper evaluates recursive weight-sharing transformer architectures as parameter-efficient surrogate models for semiconductor thermo-mechanical reliability prediction, comparing performance, parameter count, and computational cost on small engineering datasets.
This paper proposes Unified LoRA (ULoRA), a two-parameter family of preconditioned gradient initializations for low-rank adaptation, showing that existing LoRA initialization methods are points on a continuum. The authors demonstrate that a tuned ULoRA matches or exceeds full fine-tuning on GLUE tasks with RoBERTa-base and is competitive on GSM8K with LLaMA 2-7B, and introduce ULoRA-Auto for zero-search deployment.
This paper scales a closed-loop LLM-based channel configuration search to 250 candidates per cycle, showing positive accuracy trends and improved parameter efficiency on CIFAR-100, and revealing architectural regularities in LLM-generated channel priors.
SechKAN is a novel Kolmogorov-Arnold Network architecture that uses hyperbolic secant functions as basis functions, achieving competitive performance in function fitting, PDE problems, and image classification tasks while maintaining parameter efficiency comparable to MLPs.
AsySplat proposes an asymmetric architecture that decouples geometry and appearance modeling in 3D Gaussian Splatting, achieving high efficiency for long-sequence scene modeling with nearly 800x speedup over optimization-based methods.
MultiHashFormer is a hash-based generative language model that represents each token as a unique hash signature, enabling parameter-efficient autoregression. It outperforms standard Transformer LMs at 100M, 1B, and 3B scales and supports multilingual vocabulary expansion without increasing parameters.
Fenng shares a self-media comparison between the fourth-generation WeLM-80B (80B total params, 3B activated, 3.75% activation rate) and DeepSeek-V4-Flash (284B total, 13B activated, 4.6% activation rate), with a humorous comment.
Operator Boosting is a stagewise residual-learning framework that constructs compact neural operator surrogates for PDEs by training tiny models on residual fields. It achieves accuracy comparable to or better than full-size models while reducing parameters by up to 95%, demonstrating Pareto improvements on several benchmarks.
Looped World Models introduce iterative latent state refinement through shared transformer blocks, achieving 100x parameter efficiency while adapting computational depth to prediction complexity.
A student introduces Silia, a novel transformer architecture that combines attention and FFN into a unified operation to save parameters at scales ≤10M, achieving comparable performance to GPT-2 with fewer parameters despite limited compute resources.
This paper introduces CDLinear, a block-circulant neural network layer that reduces parameter count and improves Hessian conditioning via FFT diagonalization, validated on MNIST with theoretical proofs.
User questions how Qwen's 27B dense model can outperform its 397B MoE variant, sparking discussion on MoE efficiency versus dense model quality.
Andrej Karpathy claimed to Dwarkesh Patel that a 1B-parameter model trained on ultra-clean data could match today's 1.8T-parameter frontier models, implying 1,800× effective compression.
ShadowPEFT introduces a centralized parameter-efficient fine-tuning method that uses a depth-shared shadow module to refine transformer layer representations, matching or outperforming LoRA/DoRA with comparable trainable parameters.