parameter-efficiency

Tag

Cards List
#parameter-efficiency

Weight-Space Programming Doesn't Need Transformers

arXiv cs.LG ↗ · 3d ago Cached

该论文研究在无需梯度训练的权重空间编程中,LLM 自行设计目标架构时,基于联想记忆的稀疏表结构比 Transformer 更省参数且更快,最高以约1800×更少参数取得更高测试准确率。

0 favorites 0 likes
#parameter-efficiency

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Hugging Face Daily Papers ↗ · 2026-11-04 Cached

The paper proposes Loopback Transformer, which reinterprets residual connections as memory along network depth via a write–transport–read formulation, storing block updates with geometric forgetting and content-dependent attention. At matched tokens and depth it achieves lower validation loss and higher 13-task zero-shot accuracy than Standard Transformers at 1.4B and 3B while retaining only 68% and 75% of their parameters.

0 favorites 0 likes
#parameter-efficiency

Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime

Hugging Face Daily Papers ↗ · 2026-11-02 Cached

The paper introduces Library of Layers (LoL), a transformer architecture that dynamically reorders its layers at inference time, unifying parameter reuse and sparsity tradeoffs across dense, depth-recurrent, and MoE baselines while enabling interpretable routing across reasoning, multilingual, and safety tasks.

0 favorites 0 likes
#parameter-efficiency

LSTM-Style Looped Transformers with Shared KV Caches

arXiv cs.LG ↗ · 2026-10-28 Cached

This paper studies an LSTM-style looped transformer that shares both weights and a single key-value cache across its recurrent core, cutting parameters by 40.4% at width 1024 and reducing persistent cache sets from fifteen to four. A consistency loss aligning parallel and recursive training passes removes 81-89% of the continuation-loss penalty, with weight tying proving cheapest at lower token budgets.

0 favorites 0 likes
#parameter-efficiency

I built MaRN: a PyTorch library for training neural networks through low-dimensional parameter mappings [P]

Reddit r/MachineLearning ↗ · 2026-10-09

MaRN (Mapping Networks) is a new open-source PyTorch library that trains neural networks by optimizing a compact latent representation instead of all parameters directly, showing up to 57.7× parameter reduction on MNIST with modest accuracy trade-offs. The author presents exploratory benchmarks and invites community feedback on the approach.

0 favorites 0 likes
#parameter-efficiency

DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition

arXiv cs.AI ↗ · 2026-10-09 Cached

The paper proposes DivMoE, the first framework enabling fine-grained Mixture-of-Experts upcycling from dense models via domain-specialized expert initialization and diversity-constrained routing, fixing the routing collapse and accuracy regressions that plagued prior fine-grained upcycling methods across 15 benchmarks.

0 favorites 0 likes
#parameter-efficiency

One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts

Hugging Face Daily Papers ↗ · 2026-10-08 Cached

该论文提出 reViT,通过在每个递归深度用一个共享专家库的凸组合来表示 FFN,使单个 Transformer 块以递归方式应用即可在相近推理 FLOPs 下匹配全深度视觉编码器的精度,并支持弹性深度部署与权重量级压缩。

0 favorites 0 likes
#parameter-efficiency

@arXivBangers: Closing the Loop: Practical Training Recipes for Looped Language Models New training recipes enable looped language mod…

X AI KOLs Timeline ↗ · 2026-10-04 Cached

The paper introduces practical training recipes that cut looped (recurrent) language model training from 7.7T to 310B tokens, with a 1.4B LoopLM outperforming a parameter-matched dense baseline on all 12 benchmarks, and shows how to convert pretrained dense models like Qwen3-1.7B-Base into looped ones with minimal added parameters.

0 favorites 0 likes
#parameter-efficiency

@TheAhmadOsman: Qwen 4 27B.

X AI KOLs Following ↗ · 2026-10-04

A tweet announcing or referencing Qwen 4 27B, a 27-billion-parameter model in Alibaba's Qwen series.

0 favorites 0 likes
#parameter-efficiency

SW-KAN: Kolmogorov-Arnold Networks with Stieltjes-Wigert q-Orthogonal Polynomials

arXiv cs.LG ↗ · 2026-10-03 Cached

该论文提出 SW-KAN,一种基于 Stieltjes-Wigert q-正交多项式的 Kolmogorov-Arnold 网络,通过指数-tanh 域映射和 O(N) 三递推计算,在图像分类与函数逼近任务中实现更优的精度-效率权衡。

0 favorites 0 likes
#parameter-efficiency

@MaximeRivest: 4B at a time, we could get to 10T-parameter models: a true mixture of experts. Jev showed us we can route pretty decent…

X AI KOLs Timeline ↗ · 2026-10-02 Cached

Maxime Rivest argues that 4B-parameter expert models could be combined into a true mixture-of-experts reaching 10T total parameters, enabling frontier-level performance at negligible cost on edge hardware, as small models fine-tuned on few synthetic examples already rival frontier systems.

0 favorites 0 likes
#parameter-efficiency

LoopICL: Looping a single transformer block to solve tabular tasks

arXiv cs.LG ↗ · 2026-09-30 Cached

LoopICL decouples parameter count from depth by using a single weight-tied transformer block with dual cell/row streams and a learned exit gate, matching TabICL v2 on TabArena/TALENT at the same FLOPs while using ~90% fewer parameters.

0 favorites 0 likes
#parameter-efficiency

@TechByMarkandey: What if progress is no longer about adding parameters?

X AI KOLs Timeline ↗ · 2026-09-07 Cached

MiniCPM5-2B is a 2B-parameter open-source language model that achieves high intelligence density for edge deployment and ranks highly in several AI performance benchmarks.

0 favorites 0 likes
#parameter-efficiency

NanoSleep: A Parameter-Efficient Hybrid Temporal Convolutional Network for Single-Channel Sleep Stage Classification

arXiv cs.LG ↗ · 2026-08-20 Cached

The paper presents NanoSleep, a parameter-efficient hybrid temporal convolutional network for automatic sleep stage classification using single-channel EEG, designed for wearable and home-based monitoring on resource-constrained devices.

0 favorites 0 likes
#parameter-efficiency

Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models

arXiv cs.CL ↗ · 2026-08-18 Cached

This paper presents Wiola, a 13M parameter decoder-only language model with novel components like Spiral Rotary Positional Encoding and Gated Spiral Attention to enhance parameter efficiency for small-scale, on-device language models.

0 favorites 0 likes
#parameter-efficiency

Recursive transformers for semiconductor thermo-mechanical reliability

arXiv cs.LG ↗ · 2026-07-31 Cached

This paper evaluates recursive weight-sharing transformer architectures as parameter-efficient surrogate models for semiconductor thermo-mechanical reliability prediction, comparing performance, parameter count, and computational cost on small engineering datasets.

0 favorites 0 likes
#parameter-efficiency

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

arXiv cs.LG ↗ · 2026-07-30 Cached

This paper proposes Unified LoRA (ULoRA), a two-parameter family of preconditioned gradient initializations for low-rank adaptation, showing that existing LoRA initialization methods are points on a continuum. The authors demonstrate that a tuned ULoRA matches or exceeds full fine-tuning on GLUE tasks with RoBERTa-base and is competitive on GSM8K with LLaMA 2-7B, and introduce ULoRA-Auto for zero-search deployment.

0 favorites 0 likes
#parameter-efficiency

Scaling Closed-Loop Feature Channel Configuration with LLMs

arXiv cs.LG ↗ · 2026-07-24 Cached

This paper scales a closed-loop LLM-based channel configuration search to 250 candidates per cycle, showing positive accuracy trends and improved parameter efficiency on CIFAR-100, and revealing architectural regularities in LLM-generated channel priors.

0 favorites 0 likes
#parameter-efficiency

SechKAN: Kolmogorov-Arnold Networks with Hyperbolic Secant Functions

arXiv cs.LG ↗ · 2026-07-22 Cached

SechKAN is a novel Kolmogorov-Arnold Network architecture that uses hyperbolic secant functions as basis functions, achieving competitive performance in function fitting, PDE problems, and image classification tasks while maintaining parameter efficiency comparable to MLPs.

0 favorites 0 likes
#parameter-efficiency

AsySplat: Efficient Asymmetric 3D Gaussian Splatting for Long-Sequence Scene Modeling

Hugging Face Daily Papers ↗ · 2026-07-13 Cached

AsySplat proposes an asymmetric architecture that decouples geometry and appearance modeling in 3D Gaussian Splatting, achieving high efficiency for long-sequence scene modeling with nearly 800x speedup over optimization-based methods.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback