CollectionLoRA: Collecting 50 Effects in 1 LoRA via Multi-Teacher On-Policy Distillation

Hugging Face Daily Papers Papers
lora distillation diffusion-models image-editing concept-isolation parameter-interference deployment

Summary

CollectionLoRA proposes a multi-teacher on-policy distillation framework that distills up to 50 different effect LoRAs into a single LoRA, reducing deployment overhead while maintaining concept fidelity.

Customized image editing aims to equip pre-trained diffusion models with specific visual effects using limited paired data, typically via Low-Rank Adaptation (LoRA). As the number of desired effects grows, storing and dynamically loading numerous these effect LoRAs significantly increases deployment overhead. Furthermore, current pipelines typically cascade these effect LoRAs with acceleration modules for fast generation, which triggers severe parameter interference and results in concept bleeding and style degradation. We propose CollectionLoRA, a multi-teacher on-policy distillation framework capable of distilling the concepts of up to 50 different effect LoRAs along with few-step generation capabilities into a single LoRA. This fundamentally resolves the feature interference issue and significantly reduces deployment costs. Specifically, the method introduces (i) a Probabilistic Dual-Stream Routing mechanism that enables the model to randomly switch between data sources during training, effectively enhancing its generalization in unseen scenarios; (ii) an Asymmetric Orthogonal Prompting strategy to achieve concept isolation within the prompt space; (iii) a Coarse-to-Fine Distillation Objective to mitigate the distribution gap between the teacher and student models. Extensive evaluations show that CollectionLoRA distills all customized effects and few-step generation into a single LoRA, reducing deployment overhead while achieving concept fidelity comparable to or better than independently trained teacher models.
Original Article
View Cached Full Text

Cached at: 05/29/26, 07:00 AM

Paper page - CollectionLoRA: Collecting 50 Effects in 1 LoRA via Multi-Teacher On-Policy Distillation

Source: https://huggingface.co/papers/2605.25378

Abstract

CollectionLoRA enables efficient deployment of multiple customized image editing effects by distilling numerous LoRAs into a single model through multi-teacher distillation and specialized mechanisms for concept isolation and generation.

Customized image editing aims to equip pre-traineddiffusion modelswith specific visual effects using limited paired data, typically viaLow-Rank Adaptation(LoRA). As the number of desired effects grows, storing and dynamically loading numerous these effectLoRAs significantly increases deployment overhead. Furthermore, current pipelines typically cascade these effectLoRAs with acceleration modules for fast generation, which triggers severeparameter interferenceand results inconcept bleedingandstyle degradation. We propose CollectionLoRA, amulti-teacher on-policy distillationframework capable of distilling the concepts of up to 50 different effectLoRAs along with few-step generation capabilities into a singleLoRA. This fundamentally resolves the feature interference issue and significantly reduces deployment costs. Specifically, the method introduces (i) aProbabilistic Dual-Stream Routingmechanism that enables the model to randomly switch between data sources during training, effectively enhancing its generalization in unseen scenarios; (ii) anAsymmetric Orthogonal Promptingstrategy to achieveconcept isolationwithin the prompt space; (iii) aCoarse-to-Fine Distillation Objectiveto mitigate the distribution gap between the teacher and student models. Extensive evaluations show that CollectionLoRAdistills all customized effects and few-step generation into a singleLoRA, reducing deployment overhead while achieving concept fidelity comparable to or better than independently trained teacher models.

View arXiv pageView PDFProject pageGitHub5Add to collection

Get this paper in your agent:

hf papers read 2605\.25378

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.25378 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.25378 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.25378 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

OPRD: On-Policy Representation Distillation

Hugging Face Daily Papers

OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Hugging Face Daily Papers

This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.

Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training

arXiv cs.LG

Hybrid-LoRA proposes a framework that selectively applies full fine-tuning to a small subset of modules while using LoRA for the rest, achieving performance near full fine-tuning with significantly lower computational cost. Experiments show improvements of up to 5.65% over existing parameter-efficient baselines.

On-Policy Distillation (5 minute read)

TLDR AI

This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.