Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts
Summary
This paper introduces CaRE, a novel continual learning framework using a bi-level routing mixture-of-experts mechanism to effectively handle class-incremental learning over sequences of 300+ tasks.
View Cached Full Text
Cached at: 05/11/26, 07:21 AM
Paper page - Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts
Source: https://huggingface.co/papers/2602.03473
Abstract
A novel continual learning framework called CaRE with a bi-level routing mixture-of-experts mechanism is proposed for class-incremental learning, demonstrating superior performance on very long task sequences exceeding 300 tasks.
Continual learning, especiallyclass-incremental learning(CIL), on the basis of apre-trained model(PTM) has garnered substantial research interest in recent years. However, how to effectively learn both discriminative and comprehensive feature representations while maintaining stability and plasticity over very long task sequences remains an open problem. We propose CaRE, a scalable {C}ontinual Le{a}rner with efficient Bi-Level {R}outing Mixture-of-{E}xperts (BR-MoE). The core idea of BR-MoE is abi-level routingmechanism: a router selection stage that dynamically activates relevanttask-specific routers, followed by anexpert routingphase that dynamically activates and aggregates experts, aiming to inject discriminative andcomprehensive representationsinto every intermediate network layer. On the other hand, we introduce a challenging dataset,OmniBenchmark-1K, for CIL performance evaluation on very long task sequences with hundreds of tasks. Extensive experiments show that CaRE demonstrates leading performance across a variety of datasets and task settings, including commonly used CIL datasets with classical CIL settings (e.g., 5-20 tasks). To the best of our knowledge, CaRE is the first continual learner that scales to very long task sequences (ranging from 100 to over 300 non-overlapping tasks), while outperforming all baselines by a large margin on such task sequences. We hope that this work will inspire further research intocontinual learningover extremely long task sequences. Code and dataset are publicly released at https://github.com/LMMMEng/CaRE.
View arXiv pageView PDFGitHub5Add to collection
Get this paper in your agent:
hf papers read 2602\.03473
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2602.03473 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2602.03473 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2602.03473 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Exploring continual learning without replay buffers: Our findings using dynamic task-similarity routing [P]
Introduces Coincidex, an open-source continual learning framework that uses dynamic task-similarity routing to avoid replay buffers, with benchmark results and identified failure modes.
CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents
CERA-MoA introduces a co-evolving framework for mixture-of-agents systems that uses reinforcement learning to dynamically route queries and adapt agent capabilities, enhancing task performance and efficiency.
Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
The paper proposes TRACE, a method for machine unlearning in Mixture-of-Experts language models that calibrates retain regularization by reweighting token-level retain losses to address forget-retain routing mismatch. Experiments show improved forget-utility trade-off across multiple MoE LLMs.
Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning
This paper proposes FedFMX, a Fisher-Routed Mixture of Experts framework for Federated Class-Incremental Learning, addressing capacity conflict, catastrophic forgetting, and data heterogeneity via adaptive expert specialization.
Attention-Aware Routing: Coupling Routing and Attention in MoEs
This paper introduces Attention-Aware Routing (AAR), a method that enhances Mixture-of-Experts language models by incorporating attention weights into the router, improving mathematical reasoning performance and revealing coupled dynamics between routing and attention.