Tag
该论文研究在无需梯度训练的权重空间编程中,LLM 自行设计目标架构时,基于联想记忆的稀疏表结构比 Transformer 更省参数且更快,最高以约1800×更少参数取得更高测试准确率。
The paper proposes Loopback Transformer, which reinterprets residual connections as memory along network depth via a write–transport–read formulation, storing block updates with geometric forgetting and content-dependent attention. At matched tokens and depth it achieves lower validation loss and higher 13-task zero-shot accuracy than Standard Transformers at 1.4B and 3B while retaining only 68% and 75% of their parameters.
The paper introduces Library of Layers (LoL), a transformer architecture that dynamically reorders its layers at inference time, unifying parameter reuse and sparsity tradeoffs across dense, depth-recurrent, and MoE baselines while enabling interpretable routing across reasoning, multilingual, and safety tasks.
This paper studies an LSTM-style looped transformer that shares both weights and a single key-value cache across its recurrent core, cutting parameters by 40.4% at width 1024 and reducing persistent cache sets from fifteen to four. A consistency loss aligning parallel and recursive training passes removes 81-89% of the continuation-loss penalty, with weight tying proving cheapest at lower token budgets.
MaRN (Mapping Networks) is a new open-source PyTorch library that trains neural networks by optimizing a compact latent representation instead of all parameters directly, showing up to 57.7× parameter reduction on MNIST with modest accuracy trade-offs. The author presents exploratory benchmarks and invites community feedback on the approach.
The paper proposes DivMoE, the first framework enabling fine-grained Mixture-of-Experts upcycling from dense models via domain-specialized expert initialization and diversity-constrained routing, fixing the routing collapse and accuracy regressions that plagued prior fine-grained upcycling methods across 15 benchmarks.
该论文提出 reViT,通过在每个递归深度用一个共享专家库的凸组合来表示 FFN,使单个 Transformer 块以递归方式应用即可在相近推理 FLOPs 下匹配全深度视觉编码器的精度,并支持弹性深度部署与权重量级压缩。
The paper introduces practical training recipes that cut looped (recurrent) language model training from 7.7T to 310B tokens, with a 1.4B LoopLM outperforming a parameter-matched dense baseline on all 12 benchmarks, and shows how to convert pretrained dense models like Qwen3-1.7B-Base into looped ones with minimal added parameters.
A tweet announcing or referencing Qwen 4 27B, a 27-billion-parameter model in Alibaba's Qwen series.
该论文提出 SW-KAN,一种基于 Stieltjes-Wigert q-正交多项式的 Kolmogorov-Arnold 网络,通过指数-tanh 域映射和 O(N) 三递推计算,在图像分类与函数逼近任务中实现更优的精度-效率权衡。
Maxime Rivest argues that 4B-parameter expert models could be combined into a true mixture-of-experts reaching 10T total parameters, enabling frontier-level performance at negligible cost on edge hardware, as small models fine-tuned on few synthetic examples already rival frontier systems.
LoopICL decouples parameter count from depth by using a single weight-tied transformer block with dual cell/row streams and a learned exit gate, matching TabICL v2 on TabArena/TALENT at the same FLOPs while using ~90% fewer parameters.
MiniCPM5-2B is a 2B-parameter open-source language model that achieves high intelligence density for edge deployment and ranks highly in several AI performance benchmarks.
The paper presents NanoSleep, a parameter-efficient hybrid temporal convolutional network for automatic sleep stage classification using single-channel EEG, designed for wearable and home-based monitoring on resource-constrained devices.
This paper presents Wiola, a 13M parameter decoder-only language model with novel components like Spiral Rotary Positional Encoding and Gated Spiral Attention to enhance parameter efficiency for small-scale, on-device language models.
This paper evaluates recursive weight-sharing transformer architectures as parameter-efficient surrogate models for semiconductor thermo-mechanical reliability prediction, comparing performance, parameter count, and computational cost on small engineering datasets.
This paper proposes Unified LoRA (ULoRA), a two-parameter family of preconditioned gradient initializations for low-rank adaptation, showing that existing LoRA initialization methods are points on a continuum. The authors demonstrate that a tuned ULoRA matches or exceeds full fine-tuning on GLUE tasks with RoBERTa-base and is competitive on GSM8K with LLaMA 2-7B, and introduce ULoRA-Auto for zero-search deployment.
This paper scales a closed-loop LLM-based channel configuration search to 250 candidates per cycle, showing positive accuracy trends and improved parameter efficiency on CIFAR-100, and revealing architectural regularities in LLM-generated channel priors.
SechKAN is a novel Kolmogorov-Arnold Network architecture that uses hyperbolic secant functions as basis functions, achieving competitive performance in function fitting, PDE problems, and image classification tasks while maintaining parameter efficiency comparable to MLPs.
AsySplat proposes an asymmetric architecture that decouples geometry and appearance modeling in 3D Gaussian Splatting, achieving high efficiency for long-sequence scene modeling with nearly 800x speedup over optimization-based methods.