On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
Summary
This paper explores using parameter-efficient fine-tuning (PEFT) as a compact substrate for persistent personal models, studying scaling up, down, and out, and introduces MinT for managing adapters.
View Cached Full Text
Cached at: 06/02/26, 03:23 AM
Paper page - On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
Source: https://huggingface.co/papers/2606.02437 Published on Jun 1
#1 Paper of the day Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Parameter-efficient fine-tuning can function as a compact substrate for persistent personal models by enabling small trainable adapters to store instance-specific behaviors on top of strong foundation models.
Parameter-efficient fine-tuning(PEFT) is usually treated as a cheaper alternative tofull fine-tuning. We study a broader role: smalltrainable adaptersaspersistent local stateon top of strongshared foundation models. In this framing, the base model provides shared competence while adapters carryinstance-specific behaviorsuch as preferences, skills, tool habits, and memory-like updates. We organize the problem around three scaling axes: Scale Up, where strongershared priorsmake small local updates more useful; Scale Down, where we study how small adapters can be while remaining reliable; and Scale Out, where many persistent adapted instances coexist. MinT provides one infrastructure example for managingadapter identity,revision,provenance,evaluation, andserving residency. Together, the results suggest that PEFT can be a compact substrate for persistent personal models rather than only a budget substitute forfull fine-tuning.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.02437 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.02437 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.02437 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Fine-Tuning of Transformer models with Frames
The paper proposes FrameFT, a parameter-efficient fine-tuning method using sparse coefficients in a Fusion Frame basis, reducing memory footprint while achieving performance on par with or exceeding state-of-the-art PEFT techniques.
ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
ShadowPEFT introduces a centralized parameter-efficient fine-tuning method that uses a depth-shared shadow module to refine transformer layer representations, matching or outperforming LoRA/DoRA with comparable trainable parameters.
The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning
This paper benchmarks sub-1B models on mathematical reasoning tasks, revealing that full fine-tuning actively harms performance in models under 300M parameters, while parameter-efficient fine-tuning (PEFT) like LoRA and DoRA provides stability. The authors recommend defaulting to PEFT for all aligned sub-1B models and caution against full FT for architectures smaller than 500M to prevent catastrophic forgetting.
Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs
The paper compares five parameter-efficient fine-tuning methods on four small language models for on-device personalization, finding LoRA+ best for energy efficiency and QLoRA best for memory-limited deployment.
@_vmlops: FINE-TUNING A 12B MODEL ON A SINGLE GPU IS REAL NOW most people think you need a massive gpu cluster to fine-tune large…
Hugging Face's PEFT library enables parameter-efficient fine-tuning of large models on a single GPU, reducing compute and storage costs while maintaining performance.