parameter-efficiency

Tag

Cards List
#parameter-efficiency

MultiHashFormer: Hash-based Generative Language Models

arXiv cs.CL ↗ · 2026-06-29 Cached

MultiHashFormer is a hash-based generative language model that represents each token as a unique hash signature, enabling parameter-efficient autoregression. It outperforms standard Transformer LMs at 100M, 1B, and 3B scales and supports multilingual vocabulary expansion without increasing parameters.

0 favorites 0 likes
#parameter-efficiency

@Fenng: Saw this piece written by a self-media account — 'The latest fourth-generation WeLM-80B now has only 80 billion total parameters, with 3 billion activated, an activation rate of just 3.75%. For comparison — DeepSeek-V4-Flash, the domestic representative of extreme cost-performance, has 284 billion total parameters, 13 billion activated, activation rate of 4.6%...'

X AI KOLs Timeline ↗ · 2026-06-23 Cached

Fenng shares a self-media comparison between the fourth-generation WeLM-80B (80B total params, 3B activated, 3.75% activation rate) and DeepSeek-V4-Flash (284B total, 13B activated, 4.6% activation rate), with a humorous comment.

0 favorites 0 likes
#parameter-efficiency

Operator Boosting Produces Pareto-Efficient PDE Surrogates

arXiv cs.LG ↗ · 2026-06-17 Cached

Operator Boosting is a stagewise residual-learning framework that constructs compact neural operator surrogates for PDEs by training tiny models on residual fields. It achieves accuracy comparable to or better than full-size models while reducing parameters by up to 95%, demonstrating Pareto improvements on several benchmarks.

0 favorites 0 likes
#parameter-efficiency

Looped World Models

Hugging Face Daily Papers ↗ · 2026-06-16 Cached

Looped World Models introduce iterative latent state refinement through shared transformer blocks, achieving 100x parameter efficiency while adapting computational depth to prediction complexity.

0 favorites 0 likes
#parameter-efficiency

Tiny Scale Is All I Can Spare To Play With Transformer

Reddit r/LocalLLaMA ↗ · 2026-06-11

A student introduces Silia, a novel transformer architecture that combines attention and FFN into a unified operation to save parameters at scales ≤10M, achieving comparable performance to GPT-2 with fewer parameters despite limited compute resources.

0 favorites 0 likes
#parameter-efficiency

Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count

arXiv cs.LG ↗ · 2026-05-12 Cached

This paper introduces CDLinear, a block-circulant neural network layer that reduces parameter count and improves Hessian conditioning via FFT diagonalization, validated on MNIST with theoretical proofs.

0 favorites 0 likes
#parameter-efficiency

Forgive my ignorance but how is a 27B model better than 397B?

Reddit r/LocalLLaMA ↗ · 2026-04-22

User questions how Qwen's 27B dense model can outperform its 397B MoE variant, sparking discussion on MoE efficiency versus dense model quality.

0 favorites 0 likes
#parameter-efficiency

@aakashgupta: Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1…

X AI KOLs Timeline ↗ · 2026-04-22 Cached

Andrej Karpathy claimed to Dwarkesh Patel that a 1B-parameter model trained on ultra-clean data could match today's 1.8T-parameter frontier models, implying 1,800× effective compression.

0 favorites 0 likes
#parameter-efficiency

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

arXiv cs.CL ↗ · 2026-04-22 Cached

ShadowPEFT introduces a centralized parameter-efficient fine-tuning method that uses a depth-shared shadow module to refine transformer layer representations, matching or outperforming LoRA/DoRA with comparable trainable parameters.

0 favorites 0 likes
← Previous
← Back to home

Submit Feedback