parameter-efficient

The paper proposes LLMAE, a method to repurpose pre-trained decoder-only LLMs as continuous text autoencoders using a latent bottleneck, achieving high-fidelity reconstruction and enabling downstream tasks like image captioning.

This paper proposes modality-gated deep adapters to extend frozen multimodal embedding models with new modalities while preserving existing outputs bit-for-bit, demonstrated with audio and thermal packs that improve benchmark performance.

This paper proposes a lightweight evolutionary heuristic scheduler to optimize denoising trajectories in diffusion large language models, addressing failure modes like EOS Overflow and Proximal Bias, and outperforming baselines on reasoning and planning benchmarks.

ALPINE introduces an ultra-lightweight spatial-relational architecture for few-shot image classification that achieves accuracy gains with fewer parameters, faster convergence, and better robustness compared to baselines like Prototypical Networks and MAML.

This paper presents a three-stage training pipeline for developing compact dense retrievers, introducing PolDense for Polish and EuroDense for European languages, which achieve strong performance with significantly reduced parameters compared to larger models.

This paper introduces an adaptive anisotropic learning-rate model for LoRA fine-tuning to address within-module heterogeneity, improving performance and rank capacity utilization across benchmarks.

This paper diagnoses intra-adapter contention in MoE+LoRA fine-tuning and introduces SpawnLoRA to dynamically add sub-adapters, reducing negative transfer across domains.

DRET is a parameter-efficient knowledge-transfer method that injects biomedical domain knowledge into smaller models like DistilBERT via embedding transfer, achieving performance competitive with larger specialized models.

This paper explores the trade-off between factual knowledge acquisition and capability retention in language models when using QLoRA, showing that higher-rank QLoRA improves fact learning but may degrade out-of-domain performance.

This article explains five parameter-efficient fine-tuning techniques for large language models, such as LoRA and VeRA, detailing how each method adapts model weights with minimal updates.

The paper introduces Gated Recurrent Transformer, a model that reuses a shared core across depth with adaptive update gates, achieving comparable or better quality than deeper models with fewer parameters and lower memory.

PETA proposes a parameter-efficient framework for test-time adaptation in virtual screening, improving performance by updating only LayerNorm parameters during inference.

HiRoute proposes a hierarchical routed prompt-tuning framework that separates category-agnostic safety control from category-specific response guidance for LLM safety alignment, reducing over-refusal while maintaining high safety rates.

This paper proposes ODE-based Transformer decoders using Runge-Kutta integration methods to improve iterative refinement for sign language translation, achieving better BLEU scores than the IPSLT baseline without increasing model size.

Introduces AuroOFT, a method that enhances quantized orthogonal fine-tuning (QOFT) with a zero-start gated low-rank nonlinear residual, improving mathematical reasoning accuracy over QOFT and QLoRA while using fewer trainable parameters on low-bit language models.

AuroSFT introduces a parameter-efficient adapter-based framework for multi-task supervised fine-tuning that rolls back adapter checkpoints at task-wise peaks instead of full-model checkpoints, achieving higher average accuracy than mSFT.

Introduces LoCA, a two-stage backpropagation-free method for small-shift adaptation of LLMs, using one-shot calibration to fit local credit assignment maps and closed-form ridge solves for low-rank adapters, achieving lower memory and time than LoRA with competitive cross-entropy on multiple benchmarks.

SpecPrefetch proposes a parameter-efficient expert prefetching framework for sparse MoE models, using a lightweight adapter to predict next-layer experts for asynchronous transfer while preserving native routing semantics. It achieves up to 20% decoding throughput improvement on a Snapdragon 8 Elite device, demonstrating practical benefits for memory-constrained deployment.

Introduces TopoTuner, a topology-guided fine-tuning framework that selectively freezes attention projection matrices by measuring topological drift via Wasserstein distances between persistence diagrams. It achieves competitive performance to full fine-tuning while training only 1-2% of parameters and outperforms LoRA in most settings.

Introduces Attention Head Reweighting (AHR), a data-efficient method for adapting LLMs to text classification tasks by learning a single scalar per attention head, drastically reducing trainable parameters while outperforming LoRA in limited data settings.