Tag
This paper introduces RASA, a receiver-aware framework for redistributing aggregated LoRA updates in federated learning, showing that spectral truncation can cause arbitrarily higher response error than input-aware readouts and achieving gains such as +3.23 points on DomainNet-100 and up to +8.33 ROUGE-L on SuperNI.
Concertina 技术报告提出将教师模型的每 N 个相邻层折叠为一个共享块并执行 N 次,配合低秩修正与逐 token 门控,从教师权重直接构建学生模型。在 Qwen3-4B 上蒸馏出的 2.27B 模型仅用 22B tokens 即在八个基准上平均保留教师 84% 的能力。
The paper proposes PROLA, a principal-orthogonal low-rank adaptation method that allocates a fixed rank budget between pretrained weights' principal directions and their orthogonal complement, and introduces a Predictive Spatiotemporal Weather Downscaling task where PROLA outperforms existing low-rank adaptation baselines across diverse datasets.
提出一种层级低秩重参数化方法,将PEFT方法的可训练参数重组为近似方阵并递归进行低秩分解,可在压缩约一个数量级参数的同时保持甚至提升LoRA/DoRA在常识推理与数学推理任务上的性能。
A new arXiv paper introduces Functional Rank Allocation (FRA), an unlabeled method for compressing large fleets of LoRA adapters via functional rank-distortion frontiers and fleet-level minimax allocation under a shared rank budget, improving worst-case and tail utility over uniform spectral compression across 78 adapters from three public fleets.
提出 Self-Direction 机制,让智能体在任务失败后自我反思并将教训蒸馏进轻量级 LoRA 适配器,构建可持久复用的教训库,在 GAIA、Claw-Eval 和 Terminal-Bench 2.1 上将成功率提升 8.9 至 17.0 个百分点,并显著减少重复犯错。
论文提出“watermarkability”指标来量化语言模型与基于logit的水印规则的兼容性,并通过两阶段方法(目标集条件LoRA加闭环控制器)在保持高水印可靠性的同时显著降低困惑度增加。
ToPA 提出了一种基于分块Toeplitz矩阵的参数高效微调方法,避免了低秩和稀疏结构的限制,在21个NLP和CV数据集、5种模型架构上验证了其表达力与训练稳定性。
MorphAct 提出一种上下文参数化框架,通过两个耦合的低秩生成器动态调整机器人策略参数,缓解不同交互上下文中相互冲突的参数更新,在 LIBERO、Meta-World 及真实机器人实验中相比多个 VLA 骨干网络最高提升 16.30%。
论文提出 Cortex 方法,通过生成前瞻性查询并将上下文条件化输出蒸馏为可合并的低秩或结构化前馈网络更新,实现将上下文增益固化到参数记忆中,在保留 99.8% 原有非目标任务性能的同时恢复 88.9% 的上下文增益。
本文档研究了基于SVD的KV缓存压缩中"仅编码器修复"的参数高效微调方法,发现当为每个实验组单独调整学习率后,仅训练编码器(冻结解码器)以3倍更少的可训练参数和3倍更少的优化器内存即可达到与全量微调相同的精度,同时揭示了共享学习率在比较不同参数量微调方案时的常见实验缺陷。
MuLoRA introduces a spectrally balanced low-rank adaptation method for continual learning that identifies 'spectral plasticity collapse' in LoRA updates and addresses it via historical whitening and momentum-based orthogonalization, achieving the best mean accuracy in 15 of 16 reported metrics across class-incremental benchmarks.
This paper proposes PALM, a low-rank adapter method to adapt financial language models to specific time points without full retraining, reducing look-ahead bias and computational costs in financial backtests.
This paper introduces ℓp-LoRA, a principled method for automatic rank allocation in low-rank adaptation using ℓp regularization, demonstrating competitive performance on NLP tasks.
The paper proposes ChainDoRA, a parameter-efficient fine-tuning method for large language models that combines Tensor-Train factorization with weight-decomposed low-rank adaptation, achieving a 90.62% reduction in trainable parameters and improved accuracy over LoRA and DoRA.
This paper presents a parameter-efficient adaptation framework using LoRA and contrastive learning to enhance vision transformers for automatic target recognition in synthetic aperture sonar imagery.
LARA is a research project and PyTorch library that enables modular, composable behaviors for frozen large language models using low-rank residual adapters, allowing efficient training and inference-time blending of multiple behaviors.
The paper proposes FLoKD, an adaptive knowledge-distillation framework for federated LoRA fine-tuning of LLMs over wireless networks, reducing communication overhead by 50-65% while maintaining competitive performance.
This paper introduces CLAW, a method that uses hypernetworks to generate low-rank adapters for world models, enabling efficient online adaptation in model-based reinforcement learning with limited test-time data.
This paper presents Iso-LoRA, an optimizer that enhances LoRA by promoting even energy distribution across singular directions through spectral descent on tangent-space perturbations, improving effective rank and downstream performance in language model adaptation.