low-rank

This paper proposes a novel adversarial training method that eliminates input gradients by using low-rank Householder expansions, reducing computational cost while achieving robustness comparable to standard techniques for small perturbations.

This paper introduces a low-rank degree-two density projection method for nonparametric changepoint detection in high dimensions, using matrix mean estimation to handle distributional changes without parametric assumptions.

LoRA-Diffusion proposes a parameter-efficient fine-tuning method for diffusion-based language models by applying low-rank decomposition to the denoising trajectory rather than model weights, achieving competitive performance with only 1.2% trajectory adapter parameters.

This paper introduces Language-Conditional Dequantization (LCD), a post-hoc method that adds per-language low-rank LoRA corrections to quantized LLMs to recover multilingual performance lost during English-calibrated quantization, showing significant perplexity and accuracy improvements on non-English languages.

This paper introduces LorExperts and BTExperts, router-preserving compression methods for Mixture-of-Experts LLMs that cluster experts and represent non-dominant members as low-rank corrections, improving compression quality over prior methods like D2-MoE.

AuroSFT introduces a parameter-efficient adapter-based framework for multi-task supervised fine-tuning that rolls back adapter checkpoints at task-wise peaks instead of full-model checkpoints, achieving higher average accuracy than mSFT.

ARCHead is a packed LM-head compressor that combines quantized low-rank factors, INT4 residuals, and an activation-metric correction to reduce storage by ~3.7-3.9x while preserving perplexity, complementing existing block quantizers.

This paper introduces S4R, a low-rank KV cache compression method that combines selective token sampling, subspace construction, and sparse reconstruction to achieve up to 5× compression with near full-cache accuracy on LongBench and RULER benchmarks.

This paper shows that gently compressed LLMs can pass standard data-free quality guards (perplexity, MMLU, output fidelity) yet still invent procedure steps when used as agents, and proposes a data-free two-axis screen to detect such failures before deployment.

This paper proposes CoRA, a gradient-free framework for task-conditioned retrieval in on-device in-context learning, using frozen encoders and closed-form ridge regression to build compact retrieval bases without fine-tuning or backpropagation.

This paper introduces FLITE (Federated Low-rank Iterative Training Engine), a method for federated fine-tuning that reduces per-client communication to 1,280 floats per round (about 5KB) — an 8718× reduction over full-weight FedAvg — by using a frozen affine mapping network that generates weights from a small trainable latent and a low-rank seed-regenerable factorization, achieving accuracy within 0.5 percentage points of full-weight FedAvg on CIFAR-100 with ResNet-18.

Introduces VarRate, a training-free method for KV cache compression that allocates variable low-rank budget per token based on query salience, avoiding irreversible token eviction and outperforming uniform-rank methods at matched memory budgets on LongBench.

Proposes FedKAD, a federated Koopman anomaly detection framework for multivariate time series in IoT systems, using lightweight sliding-window Koopman representations and a Stiefel-ADMM algorithm for efficient communication and inference.

KQ-SVD is a new method for KV-cache compression that directly approximates the attention matrix via optimal low-rank decomposition, achieving 5-10x lower error than key-only SVD on LLaMA and Mistral models.

LACE-SVD is a novel low-rank compression method for large language models that uses a loss-aware rank allocation strategy and a propagation-aware correction technique to mitigate cumulative error propagation in the residual stream, achieving better perplexity than prior SVD-based methods at high compression ratios.

This paper proposes Native Factorized Weights for transformers, where every linear layer is trained as a product of two low-rank matrices from initialization. Experiments show a corpus-determined optimal rank that minimizes validation loss and a generalization band, outperforming dense baselines with fewer parameters.

Introduces Duplicated Latent Residual (DLR), a training-only, parameter-free plug-in for low-rank pre-training that improves perplexity across LLaMA models from 60M to 7B parameters, and can be folded into the model after training with zero inference cost.

This paper introduces a distributional generalization of matrix completion where each entry is a probability distribution rather than a scalar, using kernel mean embeddings and Tucker rank to capture low-rank structure. The authors propose a novel estimator with non-asymptotic error bounds and demonstrate effectiveness on synthetic and real-world data.

Introduces Eggroll, a low-rank evolution strategy for gradient-free training of spiking neural networks, reducing memory and time overhead while achieving competitive accuracy on N-MNIST.

VideoMLA replaces per-head KV caches in video diffusion models with a shared low-rank latent and decoupled 3D-RoPE positional keys, reducing per-token KV memory by 92.7% and improving throughput by 1.23x on a B200 while maintaining quality on VBench benchmarks.