Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning

# Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
Source: [https://arxiv.org/html/2607.12112](https://arxiv.org/html/2607.12112)
Jing Liu, Chenxuanyin Zou, Jiayang Ren, Gaoyun Fang, Chengfang Li, Yan Wang, Zhenchao Ma, and Bo HuJ\. Liu is with the Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada, and also with the College of Future Information Technology, Fudan University, Shanghai 200433, China \(e\-mail: jing\.liu@ieee\.org\)\. C\. Zou and J\. Ren are with the Department of Chemical and Biological Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada \(e\-mails: \{zcxy, rjy12307\}@mail\.ubc\.ca\)\. G\. Fang is with the Dyson School of Design Engineering, Royal College of Science, Imperial College London, South Kensington Campus, London SW7 2AZ, United Kingdom \(e\-mail: p\.fang23@imperial\.ac\.uk\)\. C\. Li is with the the Suzhou Institute of Biomedical Engineering and Technology \(SIBET\), Chinese Academy of Sciences, Suzhou 215163, China \(e\-mail: licf@sibet\.ac\.cn\)\. Y\. Wang is with the School of Data Science and Engineering, East China Normal University, Shanghai 200062, China \(e\-mail: yanwang@dase\.ecnu\.edu\.cn\)\. Z\. Ma is with the Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada \(e\-mail: zhenchaoma@ece\.ubc\.ca\)\. B\. Hu is with the College of Future Information Technology, Fudan University, Shanghai 200433, China \(e\-mail: bohu@fudan\.edu\.cn\)\.

###### Abstract

Federated fine\-tuning of Multimodal Large Language Models \(MLLMs\) across distributed networks enables privacy\-sensitive adaptation to evolving data streams, yet a fundamental obstacle prevents robust deployment in dynamic environments: catastrophic forgetting, wherein sequential task updates erase previously acquired knowledge across visual, linguistic, and cross\-modal representations\. Addressing this challenge is especially critical for autonomous networked AI operating in safety\-sensitive domains, such as content moderation, where reliable retention of prior knowledge underpins system integrity\. To overcome this, we propose Federated Continual Multimodal Learning \(FedCMM\), a framework that embeds continual\-learning safeguards into the federated optimization loop at three complementary levels\. At the parameter level, modality\-aware elastic weight consolidation computes separate Fisher information matrices for the vision encoder, language backbone, and cross\-modal projector, providing granular, asymmetry\-aware protection against modality\-specific forgetting\. At the data level, each client trains a lightweight local generative replay module to synthesize raw\-data\-free embedding\-level multimodal replay tuples without any raw data sharing\. At the aggregation level, Task\-similarity\-aware gradient aggregation autonomously filters and reweights client updates by gradient cosine similarity, suppressing conflicting directions and stabilizing the global learning trajectory\. Extensive experiments on two benchmarks demonstrate thatFedCMMconsistently outperforms recent baselines on accuracy and backward transfer, confirming that holistic, modality\-aware optimization enables robust evolutive adaptation across heterogeneous networked AI deployments\.

###### Index Terms:

Federated Learning, Continual Learning, Multimodal Large Language Models, Catastrophic Forgetting, Autonomous Optimization, Networked AI, Privacy Preservation

## IIntroduction

The proliferation of Multimodal Large Language Models \(MLLMs\) has marked a significant milestone in artificial intelligence, demonstrating unprecedented capabilities in understanding and generating content that integrates vision and language\[[2](https://arxiv.org/html/2607.12112#bib.bib12),[15](https://arxiv.org/html/2607.12112#bib.bib5)\]\. Driven by these capabilities, MLLMs are increasingly pivotal in high\-stakes, real\-world applications ranging from medical diagnosis and autonomous driving to content moderation in safety\-critical networked deployments\[[9](https://arxiv.org/html/2607.12112#bib.bib29),[1](https://arxiv.org/html/2607.12112#bib.bib30)\]\. However, deploying and maintaining MLLMs in privacy\-sensitive domains necessitates training methodologies that do not rely on centralized data storage\. Federated Learning \(FL\) has emerged as a leading paradigm for privacy\-preserving machine learning, enabling collaborative model training on decentralized data sources without exposing raw user data\[[17](https://arxiv.org/html/2607.12112#bib.bib13),[7](https://arxiv.org/html/2607.12112#bib.bib26),[24](https://arxiv.org/html/2607.12112#bib.bib39)\]\. Consequently, the federated fine\-tuning of MLLMs represents a critical frontier for developing adaptable and secure autonomous AI systems that can be continuously improved in dynamic network environments\[[29](https://arxiv.org/html/2607.12112#bib.bib6)\]\.

A fundamental challenge arises when these decentralized data streams are non\-stationary, evolving over time as new concepts, tasks, or data distributions emerge\. In such scenarios, MLLMs must be updated sequentially to stay current, a process known as continual learning that is fraught with the peril of catastrophic forgetting, wherein the model’s performance on previously learned tasks deteriorates drastically upon learning new ones\[[19](https://arxiv.org/html/2607.12112#bib.bib14),[28](https://arxiv.org/html/2607.12112#bib.bib35)\]\. A comprehensive taxonomy of continual learning strategies has been established\[[4](https://arxiv.org/html/2607.12112#bib.bib36)\], yet most approaches were designed for centralized settings and do not address the federated, multimodal regime\. The problem is particularly acute in safety\-critical applications: a model proficient at detecting known forms of hateful content\[[9](https://arxiv.org/html/2607.12112#bib.bib29)\]must be updated to recognize new evolving patterns, while a disaster\-response classifier must retain prior event categories as new crisis types emerge\[[1](https://arxiv.org/html/2607.12112#bib.bib30)\]\. Addressing catastrophic forgetting within a federated framework is therefore not merely a technical curiosity but a practical necessity for building robust and reliable autonomous MLLMs\.

The intersection of continual learning and federated learning introduces a complex set of technical barriers\. At its core, catastrophic forgetting stems from the stability\-plasticity dilemma, where a model must be plastic enough to acquire new knowledge yet stable enough to retain old information\[[18](https://arxiv.org/html/2607.12112#bib.bib15)\]\. Foundational continual learning methods developed for centralized settings, such as regularization\-based approaches like Elastic Weight Consolidation \(EWC\)\[[10](https://arxiv.org/html/2607.12112#bib.bib16)\]and Synaptic Intelligence\[[35](https://arxiv.org/html/2607.12112#bib.bib23)\], or replay\-based strategies that rehearse on stored exemplars\[[16](https://arxiv.org/html/2607.12112#bib.bib17)\], offer partial solutions\. However, their direct application in FL is problematic: regularization methods often struggle to scale in the complex parameter spaces of MLLMs\[[5](https://arxiv.org/html/2607.12112#bib.bib41)\], while replay\-based methods conflict with the core privacy principles of FL if raw data is stored or shared\. Moreover, the inherent statistical heterogeneity \(non\-IID data\) across clients in FL exacerbates forgetting, as divergent local updates can destructively interfere with one another during aggregation\[[36](https://arxiv.org/html/2607.12112#bib.bib18),[3](https://arxiv.org/html/2607.12112#bib.bib38)\]\.

Practical implementation challenges further compound these issues, particularly for MLLMs\. A key difficulty lies in the multi\-faceted nature of knowledge within these models: information is encoded not just within the vision encoder and language model separately but also in the intricate cross\-modal alignment modules, which introduce unique vulnerability to modality\-specific forgetting and adversarial drift\[[12](https://arxiv.org/html/2607.12112#bib.bib7)\]\. A continual learning strategy must therefore be modality\-aware, protecting critical parameters across all three components without stifling the model’s ability to learn new cross\-modal relationships\. Furthermore, client drift becomes substantially more pronounced in a continual learning context\[[26](https://arxiv.org/html/2607.12112#bib.bib8)\]: in a given round, some clients might be training on a new task while others remain on older data, producing gradient updates that are misaligned in their objectives\. Adaptive server\-side optimization methods\[[23](https://arxiv.org/html/2607.12112#bib.bib27),[8](https://arxiv.org/html/2607.12112#bib.bib28)\]reduce drift in static settings but are not designed for the evolving task objectives of continual learning\. Client\-side generative replay\[[27](https://arxiv.org/html/2607.12112#bib.bib22)\]can partially alleviate data\-level forgetting, yet synchronizing it across tasks in a privacy\-preserving federated regime remains an open challenge\.

Recent advancements in Federated Continual Learning \(FCL\) have begun to tackle some of these issues\. Methods such as FedWeIT\[[34](https://arxiv.org/html/2607.12112#bib.bib19)\]and Learning without Forgetting \(LwF\)\[[14](https://arxiv.org/html/2607.12112#bib.bib20)\]adapt knowledge distillation techniques to the federated setting, using the global model’s outputs on new data as soft labels to preserve old knowledge\. Others directly apply regularization by computing parameter importance on the server, while FedProx\-style methods\[[13](https://arxiv.org/html/2607.12112#bib.bib21)\]penalize deviations from the global model\. More recent studies have expanded the landscape: AF\-FCL selectively reuses prior knowledge under heterogeneous task drift\[[30](https://arxiv.org/html/2607.12112#bib.bib10)\], FedCBDR improves replay balance for federated class\-incremental learning\[[20](https://arxiv.org/html/2607.12112#bib.bib11)\], cooperative multi\-model training across heterogeneous devices addresses system heterogeneity from a signal\-processing perspective\[[32](https://arxiv.org/html/2607.12112#bib.bib40)\], and Sec\-MMFL studies privacy leakage and modality\-specific protection in multimodal federated learning\[[31](https://arxiv.org/html/2607.12112#bib.bib9)\]\. Although these methods mark clear progress, they are still not designed for continual federated adaptation of large MLLMs with modality\-aware consolidation, privacy\-preserving multimodal replay, and aggregation\-time conflict suppression handled in one unified pipeline\.

The limitations of existing methods motivate a new approach that synergistically integrates solutions to the core challenges of multimodal forgetting, federated data heterogeneity, and privacy exposure\. Our key insight is that an effective autonomous optimization framework for networked MLLMs must operate at three distinct levels\. At the parameter level, modality\-aware regularization protects established knowledge through parameter importance estimation\[[10](https://arxiv.org/html/2607.12112#bib.bib16)\]applied to parameter\-efficient adapter representations\[[6](https://arxiv.org/html/2607.12112#bib.bib34)\]\. At the data level, a raw\-data\-free rehearsal mechanism actively combats forgetting, extending exemplar replay\[[22](https://arxiv.org/html/2607.12112#bib.bib24)\]to a local generative synthesis paradigm suitable for federated LLM adaptation\[[33](https://arxiv.org/html/2607.12112#bib.bib37)\]\. At the aggregation level, a similarity\-driven autonomous strategy mitigates interference from heterogeneous client updates, moving beyond static parameter isolation\[[25](https://arxiv.org/html/2607.12112#bib.bib25)\]toward gradient\-informed coordination\. By combining these three pillars, it becomes possible to balance stability and plasticity in the complex, distributed, and evolutive training environment of MLLMs deployed across networked AI infrastructures\.

To address these challenges, we propose Federated Continual Multimodal Learning \(FedCMM\), a comprehensive framework designed to enable MLLMs to learn a sequence of tasks in a federated setting while mitigating catastrophic forgetting\. The core innovation ofFedCMMlies in its tripartite architecture that harmonizes parameter\-level regularization, data\-level rehearsal, and server\-level aggregation\. First, we introduce Modality\-Aware Elastic Weight Consolidation \(MA\-EWC\), a novel regularization strategy that computes and protects critical parameters independently for the vision encoder, the language model, and the cross\-modal projector\. Granular per\-modality protection prevents destructive interference\. Second, we design a Privacy\-Preserving Federated Replay \(PPFR\) mechanism, where each client trains a lightweight, local generative replay module to synthesize raw\-data\-free embedding\-level replay tuples from past tasks\. During subsequent training, clients use these replay tuples for rehearsal, effectively refreshing old knowledge without sharing any raw data\. Third, we develop Task\-Similarity\-aware Gradient Aggregation \(TSGA\), a server\-side algorithm that weights client model updates based on their gradient cosine similarity, down\-weighting updates that are likely to conflict with the current learning trajectory and thus reducing inter\-client interference\.

Our work offers several key contributions:

- •We proposeFedCMM, a federated continual multimodal learning framework designed to mitigate catastrophic forgetting during privacy\-preserving sequential MLLM fine\-tuning\.
- •We introduce three complementary components: MA\-EWC for modality\-specific parameter protection, privacy\-preserving federated replay for synthetic rehearsal, and TSGA for similarity\-aware aggregation\.
- •We conduct extensive experiments on two challenging continual federated benchmarks that show thatFedCMMoutperforms strong federated continual learning baselines\.

The remainder of this paper is organized as follows\.[Sec\.˜II](https://arxiv.org/html/2607.12112#S2)reviews related work\.[Sec\.˜IV](https://arxiv.org/html/2607.12112#S4)presents the technical details of our proposedFedCMMframework\.[Sec\.˜V](https://arxiv.org/html/2607.12112#S5)presents the experimental setup, main results, and analysis\. Finally,[Sec\.˜VI](https://arxiv.org/html/2607.12112#S6)concludes the paper and discusses future work\.

## IIRelated Work

### II\-ARegularization and Replay in Continual Learning

Continual Learning \(CL\) aims to enable models to learn from a continuous stream of data without catastrophically forgetting previously acquired knowledge\[[19](https://arxiv.org/html/2607.12112#bib.bib14)\]\. The core challenge is balancing the stability required to retain old knowledge with the plasticity needed to acquire new information\[[18](https://arxiv.org/html/2607.12112#bib.bib15)\]\. Research in CL is extensive and can be broadly categorized into three main families\. Regularization\-based methods, such as EWC\[[10](https://arxiv.org/html/2607.12112#bib.bib16)\]and Synaptic Intelligence \(SI\)\[[35](https://arxiv.org/html/2607.12112#bib.bib23)\], introduce a penalty term into the loss function to constrain updates to parameters deemed important for past tasks\. While effective, these methods can be computationally expensive and may underperform under significant task distribution shifts\. In contrast, replay\-based methods store a small buffer of exemplars from past tasks for rehearsal during the learning of new tasks\[[16](https://arxiv.org/html/2607.12112#bib.bib17)\]\. Approaches like Gradient Episodic Memory \(GEM\)\[[16](https://arxiv.org/html/2607.12112#bib.bib17)\]and iCaRL\[[22](https://arxiv.org/html/2607.12112#bib.bib24)\]have demonstrated strong performance, but their memory requirements and, more critically, the privacy implications of storing raw data make them unsuitable for many real\-world applications\. A third category, parameter\-isolation methods, allocates distinct subsets of model parameters for different tasks to prevent interference\[[25](https://arxiv.org/html/2607.12112#bib.bib25)\]\. However, such methods often suffer from limited scalability and poor knowledge transfer between tasks\. All these foundational CL strategies were designed for centralized training and cannot be directly applied in a federated setting due to privacy and communication constraints\.

### II\-BOptimization Under Heterogeneity in Federated Learning

FL provides a framework for training models on decentralized data while preserving user privacy\[[7](https://arxiv.org/html/2607.12112#bib.bib26)\]\. The seminal Federated Averaging \(FedAvg\) algorithm\[[17](https://arxiv.org/html/2607.12112#bib.bib13)\]introduced a simple yet effective procedure of local client training followed by server\-side model averaging\. A primary challenge in FL is managing statistical heterogeneity \(non\-IID data\) across clients, which can cause client drift and lead to poor convergence and performance\[[36](https://arxiv.org/html/2607.12112#bib.bib18)\]\. A significant body of work has sought to address client drift\. One line of research focuses on client\-side regularization; FedProx\[[13](https://arxiv.org/html/2607.12112#bib.bib21)\], for instance, adds a proximal term to the local loss function to limit the divergence of local models from the global model\. Another popular approach involves improving the server\-side aggregation algorithm\. For example, FedAdam and FedYogi\[[23](https://arxiv.org/html/2607.12112#bib.bib27)\]adapt concepts from adaptive optimizers to the server’s update rule to achieve faster and more stable convergence\. Similarly, SCAFFOLD\[[8](https://arxiv.org/html/2607.12112#bib.bib28)\]uses control variates to correct for client drift at both the client and server levels\. While these methods improve the robustness of FL on static, non\-IID data distributions, they inherently assume that the underlying tasks are fixed and are not designed to handle the dynamic, non\-stationary data streams encountered in continual learning scenarios\.

### II\-CFederated Continual Learning Under Task Drift

FCL has recently emerged to address the problem of learning a sequence of tasks in a federated environment\. Most existing FCL methods adapt centralized CL techniques to the constraints of FL\. For instance, LwF\-style methods\[[14](https://arxiv.org/html/2607.12112#bib.bib20)\]and FedWeIT\[[34](https://arxiv.org/html/2607.12112#bib.bib19)\]are federated adaptations of knowledge distillation, where the global model from a previous task serves as a "teacher" to regularize the training of the new model, thereby preserving old knowledge without requiring old data\. Similarly, EWC\-style methods extend regularization\-based consolidation to the federated setting\[[10](https://arxiv.org/html/2607.12112#bib.bib16)\]\. Another line of work has explored federated replay mechanisms\. Given the privacy constraints of FL, generative replay has become a promising direction\[[27](https://arxiv.org/html/2607.12112#bib.bib22)\]\. Recent work has further expanded FCL toward harder settings, including accurate forgetting under heterogeneous clients\[[30](https://arxiv.org/html/2607.12112#bib.bib10)\], online data streams\[[26](https://arxiv.org/html/2607.12112#bib.bib8)\], and balanced replay for class\-incremental federated learning\[[20](https://arxiv.org/html/2607.12112#bib.bib11)\]\. However, these approaches have predominantly been evaluated on unimodal benchmarks and do not address the modality\-specific interference, multimodal replay quality, and aggregation conflicts that arise in MLLM adaptation\. Our work fills this gap by proposing a holistic framework specifically tailored for the multimodal setting\.

## IIIPreliminaries

FL enables model training on decentralized data without requiring data centralization\. Consider a federated network ofKKclients, indexed byk∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}\. Each clientkkpossesses a local dataset𝒟k=\{\(𝐱i,yi\)\}i=1Nk\\mathcal\{D\}\_\{k\}=\\\{\(\\mathbf\{x\}\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\_\{k\}\}, whereNk=\|𝒟k\|N\_\{k\}=\|\\mathcal\{D\}\_\{k\}\|is the number of local samples\. The total number of samples in the network isN=∑k=1KNkN=\\sum\_\{k=1\}^\{K\}N\_\{k\}\. The goal of FL is to collaboratively train a single global model with parameters𝐰\\mathbf\{w\}by minimizing a global objective functionL​\(𝐰\)L\(\\mathbf\{w\}\), which is a weighted average of the local client objectivesLk​\(𝐰\)L\_\{k\}\(\\mathbf\{w\}\):

min𝐰⁡L​\(𝐰\)=∑k=1KNkN​Lk​\(𝐰\),\\displaystyle\\min\_\{\\mathbf\{w\}\}L\(\\mathbf\{w\}\)=\\sum\_\{k=1\}^\{K\}\\frac\{N\_\{k\}\}\{N\}L\_\{k\}\(\\mathbf\{w\}\),\(1\)whereLk​\(𝐰\)=1Nk​∑\(𝐱i,yi\)∈𝒟kℓ​\(f​\(𝐱i;𝐰\),yi\)L\_\{k\}\(\\mathbf\{w\}\)=\\frac\{1\}\{N\_\{k\}\}\\sum\_\{\(\\mathbf\{x\}\_\{i\},y\_\{i\}\)\\in\\mathcal\{D\}\_\{k\}\}\\ell\(f\(\\mathbf\{x\}\_\{i\};\\mathbf\{w\}\),y\_\{i\}\)is the local loss on clientkk’s data, withf​\(⋅;𝐰\)f\(\\cdot;\\mathbf\{w\}\)representing the model’s prediction andℓ​\(⋅,⋅\)\\ell\(\\cdot,\\cdot\)being a task\-specific loss function\. The canonical FL algorithm, FedAvg\[[17](https://arxiv.org/html/2607.12112#bib.bib13)\], iteratively performs the following steps: \(1\) a central server broadcasts the current global model𝐰\\mathbf\{w\}to a subset of clients; \(2\) each selected client performs multiple local training steps on its data to obtain updated local parameters𝐰k\\mathbf\{w\}\_\{k\}; \(3\) the clients send their updated parameters back to the server, which aggregates them to produce the next global model𝐰←∑kNkN​𝐰k\\mathbf\{w\}\\leftarrow\\sum\_\{k\}\\frac\{N\_\{k\}\}\{N\}\\mathbf\{w\}\_\{k\}\.

CL, also called lifelong learning, addresses the challenge of training a model on a sequence of tasks over time without access to the full data from previous tasks\. Formally, the model is exposed to a sequence ofTTtasks,𝒯=\{𝒯1,𝒯2,…,𝒯T\}\\mathcal\{T\}=\\\{\\mathcal\{T\}\_\{1\},\\mathcal\{T\}\_\{2\},\\dots,\\mathcal\{T\}\_\{T\}\\\}, where each task𝒯j\\mathcal\{T\}\_\{j\}has an associated data distributionP\(j\)​\(𝐱,y\)P^\{\(j\)\}\(\\mathbf\{x\},y\)\. The primary challenge in CL is catastrophic forgetting, where the model’s performance on earlier tasks\{𝒯1,…,𝒯j−1\}\\\{\\mathcal\{T\}\_\{1\},\\dots,\\mathcal\{T\}\_\{j\-1\}\\\}degrades significantly after being trained on the current task𝒯j\\mathcal\{T\}\_\{j\}\. The objective of CL is to learn the parameters𝐰\(T\)\\mathbf\{w\}^\{\(T\)\}for the model after the final task𝒯T\\mathcal\{T\}\_\{T\}such that the model performs well across all tasks seen so far\. A successful CL method must balance plasticity, the ability to learn new knowledge from the current task, with stability, the ability to retain knowledge from past tasks\. The federated continual multimodal learning setting integrates the challenges of FL and CL in the context of MLLMs\. In this setting, a sequence ofTTdistinct multimodal tasks arrives over time\. The data for each task𝒯j\\mathcal\{T\}\_\{j\}is distributed across theKKfederated clients, where each clientkkholds a local dataset𝒟k\(j\)\\mathcal\{D\}\_\{k\}^\{\(j\)\}\. Each data sample is a multimodal pair\(𝐱i,yi\)\(\\mathbf\{x\}\_\{i\},y\_\{i\}\), where the input𝐱i=\(𝐯i,𝐭i\)\\mathbf\{x\}\_\{i\}=\(\\mathbf\{v\}\_\{i\},\\mathbf\{t\}\_\{i\}\)consists of a visual component𝐯i∈𝒱\\mathbf\{v\}\_\{i\}\\in\\mathcal\{V\}\(e\.g\., an image\) and a textual component𝐭i∈𝒯\\mathbf\{t\}\_\{i\}\\in\\mathcal\{T\}\(e\.g\., a text prompt\)\. The modelf​\(⋅;𝐰\)f\(\\cdot;\\mathbf\{w\}\)is an MLLM parameterized by𝐰\\mathbf\{w\}\.

When the system is learning task𝒯j\\mathcal\{T\}\_\{j\}, it is assumed that data from previous tasks\{𝒯1,…,𝒯j−1\}\\\{\\mathcal\{T\}\_\{1\},\\dots,\\mathcal\{T\}\_\{j\-1\}\\\}is no longer directly available, and client data cannot be shared with the server or other clients due to privacy constraints\. The global objective at the end of learning task𝒯T\\mathcal\{T\}\_\{T\}is to find optimal model parameters𝐰∗\\mathbf\{w\}^\{\*\}that minimize the average loss across all tasks encountered:

𝐰∗=arg⁡min𝐰⁡1T​∑j=1TL\(j\)​\(𝐰\),\\displaystyle\\mathbf\{w\}^\{\*\}=\\arg\\min\_\{\\mathbf\{w\}\}\\frac\{1\}\{T\}\\sum\_\{j=1\}^\{T\}L^\{\(j\)\}\(\\mathbf\{w\}\),\(2\)whereL\(j\)​\(𝐰\)=∑k=1KNk\(j\)N\(j\)​Lk\(j\)​\(𝐰\)L^\{\(j\)\}\(\\mathbf\{w\}\)=\\sum\_\{k=1\}^\{K\}\\frac\{N\_\{k\}^\{\(j\)\}\}\{N^\{\(j\)\}\}L\_\{k\}^\{\(j\)\}\(\\mathbf\{w\}\)is the global loss for task𝒯j\\mathcal\{T\}\_\{j\}\. The core challenge is to minimize the objective in[Eq\.˜2](https://arxiv.org/html/2607.12112#S3.E2)under the dual constraints of \(1\) not having direct access to past task data and \(2\) adhering to the privacy\-preserving communication protocol of federated learning\.

## IVMethod

![Refer to caption](https://arxiv.org/html/2607.12112v1/x1.png)Figure 1:Overview of the FedCMM framework for privacy\-preserving federated continual multimodal learning\. For task𝒯j\\mathcal\{T\}\_\{j\}and communication roundrr, the server broadcasts𝐰r−1\(j−1\)\\mathbf\{w\}^\{\(j\-1\)\}\_\{r\-1\}to selected clients\. Each client trains on𝒟k\(j\)∪ℬk\(j−1\)\\mathcal\{D\}\_\{k\}^\{\(j\)\}\\cup\\mathcal\{B\}\_\{k\}^\{\(j\-1\)\}, uses MA\-EWC to protect modality\-specific parameters𝐰V\\mathbf\{w\}\_\{V\},𝐰L\\mathbf\{w\}\_\{L\}, and𝐰P\\mathbf\{w\}\_\{P\}, and uploads the local updateΔ​𝐰k\\Delta\\mathbf\{w\}\_\{k\}\. The server then computes TSGA scoressks\_\{k\}, filters conflicting updates, aggregates accepted clients with weightsαk\\alpha\_\{k\}to formΔ​𝐰r\(j\)\\Delta\\mathbf\{w\}^\{\(j\)\}\_\{r\}, and broadcasts the updated global model for the next round and task, with replay buffers and continual task evolution maintained locally\.### IV\-AMethod Overview

To address the challenge of catastrophic forgetting in federated continual learning for MLLMs, we proposeFedCMM, a framework that integrates three synergistic components to balance knowledge stability and plasticity\. As illustrated in[Fig\.˜1](https://arxiv.org/html/2607.12112#S4.F1),FedCMMembeds continual\-learning safeguards directly into the federated training loop\. Each client combines current\-task data with synthetic replay samples, constrains important modality\-specific parameters through MA\-EWC, and uploads only its local model update\. The server then evaluates update consistency through TSGA, suppresses conflicting directions, and broadcasts the aggregated global model for the next round and task\. Formally, when learning thejj\-th task𝒯j\\mathcal\{T\}\_\{j\}, each participating clientkkaims to find local model parameters𝐰k\(j\)\\mathbf\{w\}\_\{k\}^\{\(j\)\}that minimize a composite loss function on its local data𝒟k\(j\)\\mathcal\{D\}\_\{k\}^\{\(j\)\}and a set of locally generated embedding\-level replay tuplesℬk\(j−1\)\\mathcal\{B\}\_\{k\}^\{\(j\-1\)\}from the previous task\. The local objective for clientkkis:

ℒk\(j\)​\(𝐰\)=ℒtask\(j\)​\(𝐰\)\+γ​ℒreplay\(j−1\)​\(𝐰\)\+λ​ℒMA\-EWC,k\(j−1\)​\(𝐰\)\\displaystyle\\mathcal\{L\}\_\{k\}^\{\(j\)\}\(\\mathbf\{w\}\)=\\mathcal\{L\}\_\{\\text\{task\}\}^\{\(j\)\}\(\\mathbf\{w\}\)\+\\gamma\\,\\mathcal\{L\}\_\{\\text\{replay\}\}^\{\(j\-1\)\}\(\\mathbf\{w\}\)\+\\lambda\\,\\mathcal\{L\}\_\{\\text\{MA\-EWC\},k\}^\{\(j\-1\)\}\(\\mathbf\{w\}\)

\(3\)whereℒtask\(j\)\\mathcal\{L\}\_\{\\text\{task\}\}^\{\(j\)\}is the supervised loss on current\-task real samples,ℒreplay\(j−1\)\\mathcal\{L\}\_\{\\text\{replay\}\}^\{\(j\-1\)\}is the embedding\-level rehearsal loss \(see[Eq\.˜6](https://arxiv.org/html/2607.12112#S4.E6)\),γ\\gammaweights the replay contribution, andλ\\lambdabalances the MA\-EWC regularization\. After local training, the server aggregates the resulting client updates using the TSGA mechanism to produce the new global model𝐰\(j\)\\mathbf\{w\}^\{\(j\)\}\. For clarity, we explicitly partition the trainable parameters as

𝐰=𝐰V∪𝐰L∪𝐰P,𝐰V∩𝐰L=𝐰V∩𝐰P=𝐰L∩𝐰P=∅\\displaystyle\\mathbf\{w\}=\\mathbf\{w\}\_\{V\}\\cup\\mathbf\{w\}\_\{L\}\\cup\\mathbf\{w\}\_\{P\},\\quad\\mathbf\{w\}\_\{V\}\\cap\\mathbf\{w\}\_\{L\}=\\mathbf\{w\}\_\{V\}\\cap\\mathbf\{w\}\_\{P\}=\\mathbf\{w\}\_\{L\}\\cap\\mathbf\{w\}\_\{P\}=\\emptyset

\(4\)where the three subsets correspond to the vision encoder, the language backbone, and the cross\-modal projector\. In the MLLM forward pass, the vision encoder extracts visual representations𝐡V\\mathbf\{h\}\_\{V\}, which are mapped by the cross\-modal projector into the language embedding space as projected tokens𝐡^V\\hat\{\\mathbf\{h\}\}\_\{V\}\. Specifically, the projected tokens are concatenated with text token embeddings𝐡T\\mathbf\{h\}\_\{T\}and jointly processed by the language backbone to produce the output logits\. Since we employ Low\-Rank Adaptation \(LoRA\)\[[6](https://arxiv.org/html/2607.12112#bib.bib34)\]for parameter\-efficient fine\-tuning \(PEFT\),𝐰V\\mathbf\{w\}\_\{V\},𝐰L\\mathbf\{w\}\_\{L\}, and𝐰P\\mathbf\{w\}\_\{P\}refer specifically to the trainable LoRA adapter parameters within each module, while the dense pre\-trained backbone weights are kept frozen\. The task and replay contributions to[Eq\.˜3](https://arxiv.org/html/2607.12112#S4.E3)are:

ℒtask\(j\)​\(𝐰\)\\displaystyle\\mathcal\{L\}\_\{\\text\{task\}\}^\{\(j\)\}\(\\mathbf\{w\}\)=𝔼\(𝐱,y\)∼𝒟k\(j\)​\[ℓ​\(f​\(𝐱;𝐰\),y\)\],\\displaystyle=\\mathbb\{E\}\_\{\(\\mathbf\{x\},y\)\\sim\\mathcal\{D\}\_\{k\}^\{\(j\)\}\}\[\\ell\(f\(\\mathbf\{x\};\\mathbf\{w\}\),y\)\],\(5\)ℒreplay\(j−1\)​\(𝐰\)\\displaystyle\\mathcal\{L\}\_\{\\text\{replay\}\}^\{\(j\-1\)\}\(\\mathbf\{w\}\)=𝔼\(𝐡^V,𝐡^T,y^\)∼ℬk\(j−1\)​\[ℓ​\(femb​\(𝐡^V,𝐡^T;𝐰\),y^\)\],\\displaystyle=\\mathbb\{E\}\_\{\(\\hat\{\\mathbf\{h\}\}\_\{V\},\\hat\{\\mathbf\{h\}\}\_\{T\},\\hat\{y\}\)\\sim\\mathcal\{B\}\_\{k\}^\{\(j\-1\)\}\}\[\\ell\(f\_\{\\mathrm\{emb\}\}\(\\hat\{\\mathbf\{h\}\}\_\{V\},\\hat\{\\mathbf\{h\}\}\_\{T\};\\mathbf\{w\}\),\\hat\{y\}\)\],\(6\)Heref​\(𝐱;𝐰\)f\(\\mathbf\{x\};\\mathbf\{w\}\)is the standard MLLM forward on raw image\-text inputs, whilefemb​\(𝐡^V,𝐡^T;𝐰\)f\_\{\\mathrm\{emb\}\}\(\\hat\{\\mathbf\{h\}\}\_\{V\},\\hat\{\\mathbf\{h\}\}\_\{T\};\\mathbf\{w\}\)injects the generated embeddings directly at the visual\-token and text\-prompt level, bypassing the input encoder;γ\\gammaweights the replay contribution\.

### IV\-BModality\-Aware Elastic Weight Consolidation

A primary challenge in applying continual learning techniques to MLLMs is the complex and modular nature of their architecture\. Knowledge is distributed across a vision encoder, a language model, and a cross\-modal projector that aligns the two modalities\. Standard regularization methods like Elastic Weight Consolidation \(EWC\)\[[10](https://arxiv.org/html/2607.12112#bib.bib16)\]treat all model parameters uniformly\. A uniform penalty is suboptimal because it overlooks that the importance of parameters can vary significantly across different modalities\. Forgetting might occur in the visual understanding part while the linguistic part remains stable, or vice\-versa\.

To address this, we introduce Modality\-Aware EWC \(MA\-EWC\), which applies a more granular regularization by treating each core module of the MLLM independently\. The total set of trainable parameters𝐰\\mathbf\{w\}is partitioned into three disjoint subsets: vision parameters𝐰V\\mathbf\{w\}\_\{V\}, language parameters𝐰L\\mathbf\{w\}\_\{L\}, and cross\-modal projector parameters𝐰P\\mathbf\{w\}\_\{P\}\. For each subset, clientkkcomputes a separate diagonal Fisher Information Matrix \(FIM\) at the end of learning task𝒯j−1\\mathcal\{T\}\_\{j\-1\}\. The FIM serves as a proxy for parameter importance, with larger diagonal values indicating parameters that are more critical for performance on past tasks\. The client\-specific FIM for a parameter set𝐰M\\mathbf\{w\}\_\{M\}\(whereM∈\{V,L,P\}M\\in\\\{V,L,P\\\}\) is approximated as:

𝐅M,k\(j−1\)=𝔼\(𝐱,y\)∼𝒟k\(j−1\)​\[\(∇𝐰Mlog⁡p​\(y\|𝐱;𝐰M\(j−1\)⁣∗\)\)2\],\\displaystyle\\resizebox\{390\.25534pt\}\{\}\{$\\displaystyle\\mathbf\{F\}\_\{M,k\}^\{\(j\-1\)\}=\\mathbb\{E\}\_\{\(\\mathbf\{x\},y\)\\sim\\mathcal\{D\}\_\{k\}^\{\(j\-1\)\}\}\\left\[\\left\(\\nabla\_\{\\mathbf\{w\}\_\{M\}\}\\log p\(y\|\\mathbf\{x\};\\mathbf\{w\}\_\{M\}^\{\(j\-1\)\*\}\)\\right\)^\{2\}\\right\]$\},\(7\)At the element level, the diagonal approximation is accumulated sample\-wise as

FM,k,i\(j−1\)≈1\|𝒟k\(j−1\)\|​∑\(𝐱,y\)∈𝒟k\(j−1\)\(∂log⁡p​\(y∣𝐱;𝐰M\(j−1\)⁣∗\)∂wM,i\)2,\\displaystyle\\resizebox\{390\.25534pt\}\{\}\{$\\displaystyle F\_\{M,k,i\}^\{\(j\-1\)\}\\approx\\frac\{1\}\{\|\\mathcal\{D\}\_\{k\}^\{\(j\-1\)\}\|\}\\sum\_\{\(\\mathbf\{x\},y\)\\in\\mathcal\{D\}\_\{k\}^\{\(j\-1\)\}\}\\left\(\\frac\{\\partial\\log p\(y\\mid\\mathbf\{x\};\\mathbf\{w\}\_\{M\}^\{\(j\-1\)\*\}\)\}\{\\partial w\_\{M,i\}\}\\right\)^\{2\}$\},\(8\)where𝐰M\(j−1\)⁣∗\\mathbf\{w\}\_\{M\}^\{\(j\-1\)\*\}are the optimal parameters for moduleMMafter task𝒯j−1\\mathcal\{T\}\_\{j\-1\}\. In the federated setting, each clientkkindependently computes and stores its own modular FIM𝐅M,k\(j−1\)\\mathbf\{F\}\_\{M,k\}^\{\(j\-1\)\}over its local dataset𝒟k\(j−1\)\\mathcal\{D\}\_\{k\}^\{\(j\-1\)\}and retains the corresponding reference parameters𝐰M,k\(j−1\)⁣∗\\mathbf\{w\}\_\{M,k\}^\{\(j\-1\)\*\}, consistent with the per\-client notation used in[Alg\.˜1](https://arxiv.org/html/2607.12112#alg1)\. The MA\-EWC regularization loss is then the sum of the penalties for each module:

ℒMA\-EWC,k\(j−1\)​\(𝐰\)=12​∑M∈\{V,L,P\}∑i∈𝐰MFM,k,i\(j−1\)​\(wi−wi\(j−1\)⁣∗\)2,\\displaystyle\\resizebox\{368\.57964pt\}\{\}\{$\\displaystyle\\mathcal\{L\}\_\{\\text\{MA\-EWC\},k\}^\{\(j\-1\)\}\(\\mathbf\{w\}\)=\\frac\{1\}\{2\}\\sum\_\{M\\in\\\{V,L,P\\\}\}\\sum\_\{i\\in\\mathbf\{w\}\_\{M\}\}F\_\{M,k,i\}^\{\(j\-1\)\}\(w\_\{i\}\-w\_\{i\}^\{\(j\-1\)\*\}\)^\{2\}$\},\(9\)which can be rewritten as the sum of modality\-specific penalties:

ℒMA\-EWC,k\(j−1\)​\(𝐰\)=ℒV,k\(j−1\)​\(𝐰V\)\+ℒL,k\(j−1\)​\(𝐰L\)\+ℒP,k\(j−1\)​\(𝐰P\),\\displaystyle\\resizebox\{368\.57964pt\}\{\}\{$\\displaystyle\\mathcal\{L\}\_\{\\text\{MA\-EWC\},k\}^\{\(j\-1\)\}\(\\mathbf\{w\}\)=\\mathcal\{L\}\_\{V,k\}^\{\(j\-1\)\}\(\\mathbf\{w\}\_\{V\}\)\+\\mathcal\{L\}\_\{L,k\}^\{\(j\-1\)\}\(\\mathbf\{w\}\_\{L\}\)\+\\mathcal\{L\}\_\{P,k\}^\{\(j\-1\)\}\(\\mathbf\{w\}\_\{P\}\)$\},\(10\)where

ℒM,k\(j−1\)​\(𝐰M\)=12​∑i∈𝐰MFM,k,i\(j−1\)​\(wi−wi\(j−1\)⁣∗\)2\.\\displaystyle\\mathcal\{L\}\_\{M,k\}^\{\(j\-1\)\}\(\\mathbf\{w\}\_\{M\}\)=\\frac\{1\}\{2\}\\sum\_\{i\\in\\mathbf\{w\}\_\{M\}\}F\_\{M,k,i\}^\{\(j\-1\)\}\(w\_\{i\}\-w\_\{i\}^\{\(j\-1\)\*\}\)^\{2\}\.\(11\)whereFM,k,i\(j−1\)F\_\{M,k,i\}^\{\(j\-1\)\}is theii\-th diagonal element of the client\-specific FIM for moduleMM\. By separating the FIM computation, MA\-EWC can, for example, heavily penalize changes to critical vision parameters if the new task is visually similar to a past one, while allowing more flexibility in the language parameters if the new task introduces novel textual concepts\. A modality\-specific approach thus provides a more effective safeguard against the nuanced forms of forgetting\.

### IV\-CPrivacy\-Preserving Federated Replay

Rehearsal, or replaying exemplars from past tasks, is one of the most effective strategies for mitigating catastrophic forgetting\. However, in the federated setting, storing and replaying raw data from previous tasks is infeasible due to privacy constraints and storage limitations on client devices\. To overcome this, we propose Privacy\-Preserving Federated Replay, a mechanism that leverages client\-side generative models to create synthetic embedding\-level replay tuples while avoiding raw\-data exchange\. After a clientkkcompletes its training on task𝒯j\\mathcal\{T\}\_\{j\}, it uses its local dataset𝒟k\(j\)\\mathcal\{D\}\_\{k\}^\{\(j\)\}to train a lightweight, local replay generatorGk\(j\)G\_\{k\}^\{\(j\)\}\. Importantly, PPFR does not attempt to generate raw image pixels or free\-form text tokens\. Instead, it operates in the frozen MLLM representation space and synthesizes multimodal replay tuples consisting of a visual\-token embedding, a text\-prompt prototype embedding, and a replay label\. In our implementation,Gk\(j\)G\_\{k\}^\{\(j\)\}is a compact latent\-noise module driven by Gaussian codes𝐳∼𝒩​\(𝟎,𝐈\)\\mathbf\{z\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\), a 64\-dimensional latent space, and an event\-conditioned code\. A two\-layer visual decoder maps the latent code to the MiniCPM visual\-token embedding space, while a prompt\-prototype table with a small residual MLP produces the corresponding textual embedding\. The replay label is obtained from the local event label during fitting and stored as a teacher\-smoothed soft target\.

The generator is trained locally for 5 fitting epochs at the task boundary with a combination of embedding reconstruction, image\-text alignment, teacher KL, and cross\-entropy losses\. When the next task𝒯j\+1\\mathcal\{T\}\_\{j\+1\}is introduced, clientkksamplesGk\(j\)G\_\{k\}^\{\(j\)\}to synthesize a small buffer of replay tuplesℬk\(j\)=\{\(𝐡^V,i,𝐡^T,i,y^i\)\}i=1\|ℬ\|\\mathcal\{B\}\_\{k\}^\{\(j\)\}=\\\{\(\\hat\{\\mathbf\{h\}\}\_\{V,i\},\\hat\{\\mathbf\{h\}\}\_\{T,i\},\\hat\{y\}\_\{i\}\)\\\}\_\{i=1\}^\{\|\\mathcal\{B\}\|\}\. SinceGk\(j\)G\_\{k\}^\{\(j\)\}, the generated replay buffer, and the Fisher statistics remain local, PPFR avoids raw\-data exchange, although it is not intended to provide a formal differential\-privacy guarantee\. The generated bufferℬk\(j\)\\mathcal\{B\}\_\{k\}^\{\(j\)\}is then interleaved with the new task’s data𝒟k\(j\+1\)\\mathcal\{D\}\_\{k\}^\{\(j\+1\)\}for local training\. Including these synthetic embedding\-level samples in the training process, as reflected in[Eq\.˜3](https://arxiv.org/html/2607.12112#S4.E3), allows the model to rehearse prior event knowledge while avoiding storage or transmission of raw multimodal posts\. We formalize the replay synthesis process as

ℬk\(j\)=\{\(𝐡^V,i,𝐡^T,i,y^i\)\|\(𝐡^V,i,𝐡^T,i,y^i\)∼Gk\(j\)​\(𝐳i,ci\),𝐳i∼𝒩​\(𝟎,𝐈\)\}i=1\|ℬ\|,\\displaystyle\\resizebox\{368\.57964pt\}\{\}\{$\\displaystyle\\mathcal\{B\}\_\{k\}^\{\(j\)\}=\\left\\\{\(\\hat\{\\mathbf\{h\}\}\_\{V,i\},\\hat\{\\mathbf\{h\}\}\_\{T,i\},\\hat\{y\}\_\{i\}\)\\ \\middle\|\\ \(\\hat\{\\mathbf\{h\}\}\_\{V,i\},\\hat\{\\mathbf\{h\}\}\_\{T,i\},\\hat\{y\}\_\{i\}\)\\sim G\_\{k\}^\{\(j\)\}\(\\mathbf\{z\}\_\{i\},c\_\{i\}\),\\ \\mathbf\{z\}\_\{i\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\\right\\\}\_\{i=1\}^\{\|\\mathcal\{B\}\|\}$\},\(12\)wherecic\_\{i\}is the sampled local event code used to condition replay generation\. We define the replay\-augmented dataset by

𝒟~k\(j\+1\)=𝒟k\(j\+1\)∪ℬk\(j\)\.\\displaystyle\\tilde\{\\mathcal\{D\}\}\_\{k\}^\{\(j\+1\)\}=\\mathcal\{D\}\_\{k\}^\{\(j\+1\)\}\\cup\\mathcal\{B\}\_\{k\}^\{\(j\)\}\.\(13\)

### IV\-DTask\-Similarity\-Aware Gradient Aggregation

In federated continual learning, client drift is exacerbated because the statistical heterogeneity across clients is compounded by temporal heterogeneity\. At any given time, different clients might be training on different tasks or on data that represents different facets of the same task, leading to local updates that may conflict with each other\. A naive federated averaging of these updates can lead to destructive interference, harming the global model’s performance on both old and new tasks\. To address this, we introduce TSGA, a server\-side mechanism to modulate the aggregation process\. The core idea is to identify and down\-weight client updates that are outliers, as they are likely to be either off\-topic for the current learning objective or of low quality\. After receiving the parameter updatesΔ​𝐰k=𝐰k−𝐰g\\Delta\\mathbf\{w\}\_\{k\}=\\mathbf\{w\}\_\{k\}\-\\mathbf\{w\}\_\{\\text\{g\}\}from each participating clientkkin a given round, the server first computes the average update direction:

𝐠¯=1\|𝒮\|​∑k∈𝒮Δ​𝐰k‖Δ​𝐰k‖,\\displaystyle\\bar\{\\mathbf\{g\}\}=\\frac\{1\}\{\|\\mathcal\{S\}\|\}\\sum\_\{k\\in\\mathcal\{S\}\}\\frac\{\\Delta\\mathbf\{w\}\_\{k\}\}\{\\\|\\Delta\\mathbf\{w\}\_\{k\}\\\|\},\(14\)where𝒮\\mathcal\{S\}is the set of participating clients\. The server then measures the alignment of each client’s update with this average direction using cosine similarity:

sk=Δ​𝐰k⋅𝐠¯‖Δ​𝐰k‖​‖𝐠¯‖\.\\displaystyle s\_\{k\}=\\frac\{\\Delta\\mathbf\{w\}\_\{k\}\\cdot\\bar\{\\mathbf\{g\}\}\}\{\\\|\\Delta\\mathbf\{w\}\_\{k\}\\\|\\\|\\bar\{\\mathbf\{g\}\}\\\|\}\.\(15\)Clients whose updates are poorly aligned with the consensus direction \(i\.e\., have a low similarity scoresks\_\{k\}\) are likely to be sources of interference\. We introduce a similarity thresholdτ\\tau\. Any client withsk<τs\_\{k\}<\\tauis considered an outlier and is excluded from the current round\. The filtered participant set is

𝒮′=\{k∈𝒮∣sk≥τ\},\\displaystyle\\mathcal\{S\}^\{\\prime\}=\\\{k\\in\\mathcal\{S\}\\mid s\_\{k\}\\geq\\tau\\\},\(16\)and, if this set is empty, the round falls back to𝒮′=𝒮\\mathcal\{S\}^\{\\prime\}=\\mathcal\{S\}to preserve a well\-defined aggregation step even under highly conflicting two\-client rounds\. The filtered set is then used with normalized aggregation weights

αk=Nk∑i∈𝒮′Ni,k∈𝒮′,\\displaystyle\\alpha\_\{k\}=\\frac\{N\_\{k\}\}\{\\sum\_\{i\\in\\mathcal\{S\}^\{\\prime\}\}N\_\{i\}\},\\quad k\\in\\mathcal\{S\}^\{\\prime\},\(17\)and the final global model is updated using a weighted average over only the clients in𝒮′\\mathcal\{S\}^\{\\prime\}:

𝐰g←𝐰g\+∑k∈𝒮′αk​Δ​𝐰k\.\\displaystyle\\mathbf\{w\}\_\{\\text\{g\}\}\\leftarrow\\mathbf\{w\}\_\{\\text\{g\}\}\+\\sum\_\{k\\in\\mathcal\{S\}^\{\\prime\}\}\\alpha\_\{k\}\\Delta\\mathbf\{w\}\_\{k\}\.\(18\)By filtering out potentially disruptive updates, TSGA ensures a more stable and coherent learning trajectory for the global model, improving its ability to consolidate knowledge from diverse clients in a sequential learning environment\. The completeFedCMMworkflow is summarized in[Alg\.˜1](https://arxiv.org/html/2607.12112#alg1)and[Alg\.˜2](https://arxiv.org/html/2607.12112#alg2)\. The first algorithm captures the task\-level server coordination, and the second details the client\-side rehearsal and local update procedure\.

Algorithm 1FedCMMtask\-level server coordination\.1:Server executes:

2:Initialize global model parameters

𝐰\(0\)\\mathbf\{w\}^\{\(0\)\}
3:foreach task

𝒯j\\mathcal\{T\}\_\{j\}for

j=1,…,Tj=1,\\dots,Tdo

4:foreach round

r=1,…,Rjr=1,\\dots,R\_\{j\}do

5:

𝒮r←\\mathcal\{S\}\_\{r\}\\leftarrowSelect a random subset of

CCclients

6:Broadcast

𝐰r−1\(j\)\\mathbf\{w\}^\{\(j\)\}\_\{r\-1\}to all clients in

𝒮r\\mathcal\{S\}\_\{r\}
7:Initialize empty list of updates

𝒰←\[\]\\mathcal\{U\}\\leftarrow\[\]
8:foreach client

k∈𝒮rk\\in\\mathcal\{S\}\_\{r\}in paralleldo

9:

Δ​𝐰k←ClientUpdate​\(k,𝐰r−1\(j\),𝒯j\)\\Delta\\mathbf\{w\}\_\{k\}\\leftarrow\\text\{ClientUpdate\}\(k,\\mathbf\{w\}^\{\(j\)\}\_\{r\-1\},\\mathcal\{T\}\_\{j\}\)
10:Add

Δ​𝐰k\\Delta\\mathbf\{w\}\_\{k\}to

𝒰\\mathcal\{U\}
11:endfor

12:// Aggregate with TSGA

13:

𝐠¯←1\|𝒮r\|​∑k∈𝒮rΔ​𝐰k‖Δ​𝐰k‖\\bar\{\\mathbf\{g\}\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{S\}\_\{r\}\|\}\\sum\_\{k\\in\\mathcal\{S\}\_\{r\}\}\\frac\{\\Delta\\mathbf\{w\}\_\{k\}\}\{\\\|\\Delta\\mathbf\{w\}\_\{k\}\\\|\}
14:

𝒮r′←\{k∈𝒮r∣cosine​\(Δ​𝐰k,𝐠¯\)≥τ\}\\mathcal\{S\}^\{\\prime\}\_\{r\}\\leftarrow\\\{k\\in\\mathcal\{S\}\_\{r\}\\mid\\text\{cosine\}\(\\Delta\\mathbf\{w\}\_\{k\},\\bar\{\\mathbf\{g\}\}\)\\geq\\tau\\\}
15:if

𝒮r′=∅\\mathcal\{S\}^\{\\prime\}\_\{r\}=\\emptysetthen

16:

𝒮r′←𝒮r\\mathcal\{S\}^\{\\prime\}\_\{r\}\\leftarrow\\mathcal\{S\}\_\{r\}
17:endif

18:

Δ​𝐰r\(j\)←∑k∈𝒮r′Nk∑i∈𝒮r′Ni​Δ​𝐰k\\Delta\\mathbf\{w\}^\{\(j\)\}\_\{r\}\\leftarrow\\sum\_\{k\\in\\mathcal\{S\}^\{\\prime\}\_\{r\}\}\\frac\{N\_\{k\}\}\{\\sum\_\{i\\in\\mathcal\{S\}^\{\\prime\}\_\{r\}\}N\_\{i\}\}\\Delta\\mathbf\{w\}\_\{k\}
19:

𝐰r\(j\)←𝐰r−1\(j\)\+Δ​𝐰r\(j\)\\mathbf\{w\}^\{\(j\)\}\_\{r\}\\leftarrow\\mathbf\{w\}^\{\(j\)\}\_\{r\-1\}\+\\Delta\\mathbf\{w\}^\{\(j\)\}\_\{r\}
20:endfor

21:

𝐰\(j\)←𝐰Rj\(j\)\\mathbf\{w\}^\{\(j\)\}\\leftarrow\\mathbf\{w\}^\{\(j\)\}\_\{R\_\{j\}\}
22:Server signals clients to prepare for next task:

23:foreach client

k=1,…,Kk=1,\\dots,Kin paralleldo

24:Set client model to

𝐰\(j\)\\mathbf\{w\}^\{\(j\)\}
25:Train local generator

Gk\(j\)G\_\{k\}^\{\(j\)\}on its data

𝒟k\(j\)\\mathcal\{D\}\_\{k\}^\{\(j\)\}
26:Compute FIMs

𝐅M,k\(j\)\\mathbf\{F\}\_\{M,k\}^\{\(j\)\}on data

𝒟k\(j\)\\mathcal\{D\}\_\{k\}^\{\(j\)\}
27:Store

Gk\(j\)G\_\{k\}^\{\(j\)\}and FIMs locally for use in task

𝒯j\+1\\mathcal\{T\}\_\{j\+1\}
28:endfor

29:endfor

30:return

𝐰\(T\)\\mathbf\{w\}^\{\(T\)\}

Algorithm 2FedCMMclient rehearsal and local update\.1:procedureClientUpdate\(

k,𝐰,𝒯jk,\\mathbf\{w\},\\mathcal\{T\}\_\{j\}\)

2:Load local dataset

𝒟k\(j\)\\mathcal\{D\}\_\{k\}^\{\(j\)\}for task

𝒯j\\mathcal\{T\}\_\{j\}
3:// PPFR: Generate embedding\-level replay tuples

4:if

j\>1j\>1then

5:// Load

Gk\(j−1\)G\_\{k\}^\{\(j\-1\)\}and

𝐅M,k\(j−1\)\\mathbf\{F\}\_\{M,k\}^\{\(j\-1\)\}from previous task

6:

ℬk\(j−1\)←\\mathcal\{B\}\_\{k\}^\{\(j\-1\)\}\\leftarrowGenerate replay tuples using

Gk\(j−1\)G\_\{k\}^\{\(j\-1\)\}
7:else

8:

ℬk\(j−1\)←∅\\mathcal\{B\}\_\{k\}^\{\(j\-1\)\}\\leftarrow\\emptyset
9:endif

10:// Local Training

11:Set local model to

𝐰k←𝐰\\mathbf\{w\}\_\{k\}\\leftarrow\\mathbf\{w\}
12:foreach local epoch

e=1,…,Ee=1,\\dots,Edo

13:foreach batch in

𝒟k\(j\)∪ℬk\(j−1\)\\mathcal\{D\}\_\{k\}^\{\(j\)\}\\cup\\mathcal\{B\}\_\{k\}^\{\(j\-1\)\}do

14:Compute loss

ℒk\(j\)\\mathcal\{L\}\_\{k\}^\{\(j\)\}using[Eq\.˜3](https://arxiv.org/html/2607.12112#S4.E3)with

𝐅M,k\(j−1\)\\mathbf\{F\}\_\{M,k\}^\{\(j\-1\)\}
15:Update

𝐰k\\mathbf\{w\}\_\{k\}with gradient descent

16:endfor

17:endfor

18:return

Δ​𝐰k=𝐰k−𝐰\\Delta\\mathbf\{w\}\_\{k\}=\\mathbf\{w\}\_\{k\}\-\\mathbf\{w\}
19:endprocedure

## VExperiments

Experiments were conducted on two federated continual social\-event benchmarks in which every task is an image\-text classification problem\. The evaluation follows chronological event streams so that the model must preserve earlier event knowledge while adapting to later multimodal distributions\.

### V\-ADatasets and Evaluation Metrics

Datasets and Metrics\.We use PHEME\[[11](https://arxiv.org/html/2607.12112#bib.bib1)\]and CrisisMMD\[[1](https://arxiv.org/html/2607.12112#bib.bib30)\], two social\-media event datasets that support image\-text event understanding\. PHEME contains rumor and breaking\-news discussions collected from social media, with 6,273 text posts and 2,089 associated images across seven events\. CrisisMMD contains crisis\-related tweets from natural disasters, with 16,097 text instances and 18,126 images annotated for emergency\-response categories\. Following chronological event\-incremental evaluation, events are ordered by time and grouped into three tasks with a\[2,2,3\]\[2,2,3\]class split for each dataset, yieldingT=3T=3tasks per stream\. All datasets are split into train/validation/test partitions with a 70/10/20 ratio\. Unless otherwise noted, client data is partitioned across 10 clients with a Dirichlet label\-skew distribution using concentrationα=0\.5\\alpha=0\.5; we additionally report the severe non\-IID settingα=0\.1\\alpha=0\.1\. LetAi,jA\_\{i,j\}be the accuracy on task𝒯j\\mathcal\{T\}\_\{j\}after training up to task𝒯i\\mathcal\{T\}\_\{i\}\. We adopt three standard continual learning metrics: i\)Average Accuracy \(Acc\), computed as1T​∑j=1TAT,j\\frac\{1\}\{T\}\\sum\_\{j=1\}^\{T\}A\_\{T,j\}; ii\)Backward Transfer \(BWT\), computed as1T−1​∑j=1T−1\(AT,j−Aj,j\)\\frac\{1\}\{T\-1\}\\sum\_\{j=1\}^\{T\-1\}\(A\_\{T,j\}\-A\_\{j,j\}\)to quantify forgetting; and iii\)Forward Transfer \(FWT\), computed as1T−1​∑j=2TAj−1,j\\frac\{1\}\{T\-1\}\\sum\_\{j=2\}^\{T\}A\_\{j\-1,j\}\. Higher Acc and FWT are better, and BWT is better when closer to zero\.

### V\-BBaseline Methods and Implementation Details

We compareFedCMMagainst a comprehensive set of baselines:Joint Training\(upper bound with all task data\),Fine\-tuning\(sequential lower bound\),FedAvg\-CL\[[17](https://arxiv.org/html/2607.12112#bib.bib13)\],FedProx\-CL\[[13](https://arxiv.org/html/2607.12112#bib.bib21)\],LwF\-FL\[[14](https://arxiv.org/html/2607.12112#bib.bib20)\],EWC\-FL\[[10](https://arxiv.org/html/2607.12112#bib.bib16)\],FedWeIT\[[34](https://arxiv.org/html/2607.12112#bib.bib19)\], andGEM\-FL\[[16](https://arxiv.org/html/2607.12112#bib.bib17)\]\. GEM\-FL stores raw replay exemplars and is therefore treated as a privacy\-violating rehearsal reference\. We also include AF\-FCL\[[30](https://arxiv.org/html/2607.12112#bib.bib10)\], a generative\-replay method based on accurate forgetting, and TEKNet\-FL\[[21](https://arxiv.org/html/2607.12112#bib.bib2)\], a federated adaptation of the temporal event knowledge learner\. All experiments were implemented using PyTorch and Hugging Face Transformers\. The base MLLM wasopenbmb/MiniCPM\-V\-2\.6\-int4\. We employed PEFT with LoRA\[[6](https://arxiv.org/html/2607.12112#bib.bib34)\]using rankr=8r=8and scaling factor 8\. Each dataset is trained as a 3\-task stream, with 50 federated rounds per task, 10 total clients, and 5 sampled clients per round unless otherwise stated\. Local training uses paged AdamW with learning rate2×10−52\\times 10^\{\-5\}, cosine decay, per\-device batch size 1, and 16 gradient\-accumulation steps\. Each selected client performs one local epoch per federated round\. PPFR fits only the local replay generator for 5 epochs at each task boundary, using a 64\-dimensional Gaussian latent code and an embedding\-level replay budget of\|ℬ\|=100\|\\mathcal\{B\}\|=100\. The main hyperparameters areλ=500\\lambda=500for MA\-EWC andτ=0\.5\\tau=0\.5for TSGA, selected by the sensitivity analysis\. Experiments were conducted on NVIDIA L60 GPUs\.

TABLE I:Performance comparison on 3\-task PHEME and CrisisMMD under moderate and severe non\-IID partitions\. The values in parentheses indicate the Dirichlet heterogeneity parameterα\\alpha\. Higher is better for Acc and FWT, while BWT is better when closer to zero\. Best results are highlighted inbold\.MethodPHEME \(0\.5\)PHEME \(0\.1\)CrisisMMD \(0\.5\)CrisisMMD \(0\.1\)Acc↑\\uparrowBWT↑\\uparrowFWT↑\\uparrowAcc↑\\uparrowBWT↑\\uparrowFWT↑\\uparrowAcc↑\\uparrowBWT↑\\uparrowFWT↑\\uparrowAcc↑\\uparrowBWT↑\\uparrowFWT↑\\uparrowJoint Training89\.2——89\.2——91\.4——91\.4——Fine\-tuning68\.4\-29\.10\.465\.7\-32\.40\.062\.6\-30\.80\.556\.1\-36\.80\.4FedAvg\-CL\[[17](https://arxiv.org/html/2607.12112#bib.bib13)\]73\.8\-22\.70\.868\.9\-27\.90\.470\.9\-24\.91\.065\.1\-31\.20\.7FedProx\-CL\[[13](https://arxiv.org/html/2607.12112#bib.bib21)\]75\.4\-20\.90\.770\.2\-25\.60\.572\.0\-23\.81\.367\.3\-28\.10\.6LwF\-FL\[[14](https://arxiv.org/html/2607.12112#bib.bib20)\]77\.1\-18\.71\.272\.6\-23\.50\.975\.9\-19\.61\.469\.7\-25\.21\.1EWC\-FL\[[10](https://arxiv.org/html/2607.12112#bib.bib16)\]78\.6\-16\.81\.074\.9\-20\.10\.877\.1\-18\.71\.472\.4\-22\.21\.0FedWeIT\[[34](https://arxiv.org/html/2607.12112#bib.bib19)\]80\.4\-14\.91\.576\.9\-17\.01\.279\.4\-15\.21\.773\.5\-20\.11\.6GEM\-FL\[[16](https://arxiv.org/html/2607.12112#bib.bib17)\]83\.2\-12\.01\.979\.6\-14\.51\.282\.2\-12\.02\.276\.9\-16\.01\.9AF\-FCL\[[30](https://arxiv.org/html/2607.12112#bib.bib10)\]84\.1\-10\.91\.879\.4\-14\.91\.782\.6\-11\.22\.178\.1\-14\.41\.7TEKNet\-FL\[[21](https://arxiv.org/html/2607.12112#bib.bib2)\]84\.6\-10\.22\.380\.7\-13\.01\.683\.4\-10\.52\.378\.0\-14\.72\.4FedCMM\(Ours\)86\.3\-8\.72\.081\.8\-12\.61\.984\.4\-8\.82\.579\.6\-12\.82\.1

### V\-CMain Results

[Table˜I](https://arxiv.org/html/2607.12112#S5.T1)reports the comparison under moderate and severe heterogeneity\. On PHEME withα=0\.5\\alpha=0\.5,FedCMMreaches 86\.3% Acc, improving over TEKNet\-FL by 1\.7% and over AF\-FCL by 2\.2%\. The gain is not limited to the final event group: BWT improves from \-10\.2% for TEKNet\-FL to \-8\.7%, indicating less damage to earlier events\. Under the severeα=0\.1\\alpha=0\.1split, all methods lose accuracy, but the ranking is not a simple shifted copy of the moderate case: GEM\-FL has slightly higher Acc than AF\-FCL, whereas AF\-FCL has stronger FWT\.FedCMMretains a 1\.1% Acc lead and keeps BWT 0\.4% closer to zero than TEKNet\-FL\. On CrisisMMD,FedCMMobtains 84\.4% and 79\.6% underα=0\.5\\alpha=0\.5andα=0\.1\\alpha=0\.1, respectively, below the non\-continual Joint Training upper bound but above the strongest continual baselines\. Several close comparisons are dataset\-dependent; for example, AF\-FCL is slightly stronger than TEKNet\-FL in severe CrisisMMD Acc, while TEKNet\-FL gives better FWT on severe CrisisMMD and moderate PHEME, reflecting a trade\-off between forward plasticity and forgetting resistance\. GEM\-FL is competitive because raw exemplars provide direct rehearsal, yet it stores original samples and is not a privacy\-preserving federated option\. The gaps are therefore not uniform across datasets or metrics: CrisisMMD benefits more from visually grounded evidence, while PHEME depends more on event\-specific textual shifts and stance cues\. Across both streams, the strongest improvements appear on BWT, suggesting that the proposed modules mainly reduce forgetting while preserving enough plasticity for later crisis and rumor events\.

### V\-DAblation Studies

To validate each component’s contribution, we conducted ablation studies on PHEME underα=0\.5\\alpha=0\.5\.[Table˜II](https://arxiv.org/html/2607.12112#S5.T2)removes each component individually, and[Table˜III](https://arxiv.org/html/2607.12112#S5.T3)analyzes the MA\-EWC design\.

Contribution of each component\.As shown in[Table˜II](https://arxiv.org/html/2607.12112#S5.T2), FedAvg\-CL reaches 73\.8% Acc with a BWT of \-22\.7%, indicating severe forgetting under event\-incremental PHEME\. Adding all three components increases Acc to 86\.3% and improves BWT to \-8\.7%\. The component effects are not identical across metrics\. Removing PPFR causes the largest retention loss, reducing Acc to 82\.4% and worsening BWT to \-13\.6%, which indicates that embedding replay is the main anchor for old\-event decision boundaries\. Removing MA\-EWC gives 83\.7% Acc and weaker FWT, suggesting that replay alone cannot fully stabilize modality\-specific adapters\. Removing TSGA gives 84\.5% Acc; its BWT is slightly better than removing MA\-EWC, but the full method still improves both retention and transfer\. Notably, the nonparallel pattern across metrics is expected because replay, consolidation, and aggregation filtering affect different failure modes\. The ablation pattern also shows that no single component fully explains the final performance: replay improves retention, MA\-EWC constrains module drift, and TSGA reduces server\-side interference\.

TABLE II:Result of component ablation on PHEME \(α=0\.5\\alpha=0\.5\)\.ConfigurationMA\-EWCPPFRTSGAAcc↑\\uparrowBWT↑\\uparrowFWT↑\\uparrowFedAvg\-CL\[[17](https://arxiv.org/html/2607.12112#bib.bib13)\]✗✗✗73\.8\-22\.70\.8FedCMMw/o\-MA\{\}\_\{\\text\{w/o\-MA\}\}✗✓✓83\.7\-10\.91\.7FedCMMw/o\-R\{\}\_\{\\text\{w/o\-R\}\}✓✗✓82\.4\-13\.61\.6FedCMMw/o\-G\{\}\_\{\\text\{w/o\-G\}\}✓✓✗84\.5\-9\.51\.9FedCMM✓✓✓86\.3\-8\.72\.0

Analysis of MA\-EWC\.[Table˜III](https://arxiv.org/html/2607.12112#S5.T3)shows that the Fisher penalty must respect the MLLM module structure\. Removing EWC entirely gives 82\.8% Acc and \-12\.4% BWT\. A uniform EWC penalty improves this to 84\.3%, but still trails the full modality\-aware design\. Single\-branch protection is unstable and metric\-dependent: vision\-only protection reaches 83\.1% Acc but gives relatively high FWT, while language\-only protection gives higher Acc and BWT but lower FWT\. Notably, visual stabilization appears to help early adaptation to new events, whereas language\-side protection better preserves accumulated rumor cues\. The full model separately protects the vision encoder, language backbone, and projector, reaching 86\.3% and the best BWT of \-8\.7%\. In turn, the evidence supports the claim that forgetting in social\-event MLLM adaptation is distributed across visual, textual, and cross\-modal alignment modules\. The separate Fisher design is therefore not only a stronger regularizer, but also a better match to how multimodal adapters store task\-specific evidence\.

TABLE III:Result of MA\-EWC ablation on PHEME \(α=0\.5\\alpha=0\.5\)\.ConfigurationVLPSep\.Acc↑\\uparrowBWT↑\\uparrowFWT↑\\uparrowFedCMMno\-EWC\{\}\_\{\\text\{no\-EWC\}\}✗✗✗✗82\.8\-12\.41\.5FedCMMuni\-EWC\{\}\_\{\\text\{uni\-EWC\}\}✓✓✓✗84\.3\-10\.91\.7FedCMMV\{\}\_\{\\text\{V\}\}✓✗✗✗83\.1\-11\.71\.8FedCMML\{\}\_\{\\text\{L\}\}✗✓✗✗83\.9\-10\.81\.5FedCMM✓✓✓✓86\.3\-8\.72\.0

Notes: V = Vision Encoder, L = Language Backbone, P = Projector, Sep\. = Separate Fisher Matrices\.

![Refer to caption](https://arxiv.org/html/2607.12112v1/x2.png)Figure 2:Parameter sensitivity of FedCMM on PHEME and CrisisMMD, where the four panels analyze hyperparametersλ\\lambda,τ\\tau,\|ℬ\|\|\\mathcal\{B\}\|, andCC\.
### V\-EParameter Sensitivity and Robustness

Sensitivity to hyperparameters\.[Fig\.˜2](https://arxiv.org/html/2607.12112#S5.F2)shows sensitivity analyses for four key hyperparameters\. The curves are not perfectly symmetric because the two benchmarks fail in different ways: PHEME is dominated by cross\-event semantic shift, while CrisisMMD is more affected by noisy image\-text alignment and class imbalance\. For MA\-EWC strengthλ\\lambda, both datasets peak atλ=500\\lambda=500, but the decline after the peak is sharper on CrisisMMD\. Increasingλ\\lambdafrom 500 to 2000 reduces CrisisMMD from 84\.4% to 80\.3%, whereas PHEME decreases from 86\.3% to 84\.2%\. In particular, over\-constraining adapters can suppress crisis\-event adaptation even when it protects old tasks\. For the TSGA thresholdτ\\tau, 0\.5 gives the best result, but the neighboring points are uneven:τ=0\.3\\tau=0\.3is more competitive on CrisisMMD \(83\.1%\), whereas PHEME is less sensitive atτ=0\.7\\tau=0\.7\. The replay buffer trend is also nonuniform\. Moving from no replay to\|ℬ\|=100\|\\mathcal\{B\}\|=100improves PHEME by 3\.9% and CrisisMMD by 4\.6%, but the intermediate gains differ across datasets, and increasing to 150 does not help\. Consequently, low\-quality synthetic tuples can begin to dilute current\-task supervision\. Finally, samplingC=5C=5clients per round is best, while theC=3C=3andC=10C=10gaps differ across datasets\. In aggregate, the pattern across all four parameters indicates that the default configuration is not an isolated lucky point; it lies in a stable region where consolidation, replay, and aggregation filtering remain balanced\.

Robustness to data heterogeneity\.[Fig\.˜3](https://arxiv.org/html/2607.12112#S5.F3)confirms thatFedCMMremains more stable as the client label distribution becomes skewed\. On PHEME, FedAvg\-CL decreases from 80\.6% atα=1\.0\\alpha=1\.0to 69\.4% atα=0\.1\\alpha=0\.1, whileFedCMMdecreases from 89\.7% to 81\.8%\. Notably, the gap is not constant: it is 9\.1% under the mild split, 12\.5% atα=0\.5\\alpha=0\.5, and 12\.4% under severe heterogeneity\. On CrisisMMD, FedAvg\-CL falls from 74\.0% to 65\.6%, whereasFedCMMfalls from 86\.1% to 79\.6%\. A related asymmetry appears in the degradation pattern: the moderate\-to\-severe drop is larger than the mild\-to\-moderate drop, especially for FedAvg\-CL on CrisisMMD\. Disaster events appear to have more heterogeneous visual evidence across clients\. Importantly, the proposed method does not eliminate non\-IID sensitivity; rather, it reduces the additional forgetting caused by skewed client updates\. In turn, the results support the role of TSGA as a stabilizer rather than as a substitute for better client coverage\. The remaining drop atα=0\.1\\alpha=0\.1also shows that the evaluation is not saturated, leaving visible room for stronger personalization or adaptive client sampling\.

![Refer to caption](https://arxiv.org/html/2607.12112v1/x3.png)Figure 3:Robustness to client heterogeneity on PHEME and CrisisMMD\.Robustness to task ordering\.[Table˜IV](https://arxiv.org/html/2607.12112#S5.T4)evaluates desynchronized client progression on PHEME, where 30% of clients lag one task behind at each boundary\. Notably, the setting is harsher than ordinary label skew because stale clients optimize an earlier event distribution while synchronized clients have already moved to the next group\. FedAvg\-CL drops from 73\.8% to 65\.7%, and EWC\-FL drops from 78\.6% to 71\.4%\.FedCMMdrops from 86\.3% to 82\.4%, losing less than 4%\. A smaller degradation is consistent with the interaction between TSGA and PPFR: stale updates tend to have lower agreement with the round consensus, and replay keeps the retained clients from over\-specializing to only the latest event\. Beyond a simple heterogeneity test, the asynchronous setting specifically probes whether aggregation can tolerate clients being at different points in the continual stream\. Moreover, the property is important for social\-event deployments, where clients may receive delayed event evidence or skip event groups entirely\.

TABLE IV:Accuracy of task\-order robustness on PHEME\.Δ\\Deltadenotes the accuracy drop from synchronized to asynchronous progression\.MethodSync\.↑\\uparrowAsync\.↑\\uparrow𝚫↑\\bm\{\\Delta\\uparrow\}FedAvg\-CL\[[17](https://arxiv.org/html/2607.12112#bib.bib13)\]73\.865\.7\-8\.1EWC\-FL\[[10](https://arxiv.org/html/2607.12112#bib.bib16)\]78\.671\.4\-7\.2FedCMM86\.382\.4\-3\.9

### V\-FLocal\-Epoch Sensitivity and Efficiency Analysis

Effect of local epoch numberEE\.[Fig\.˜4](https://arxiv.org/html/2607.12112#S5.F4)shows average accuracy asEEincreases from 1 to 4\. All methods peak atE=1E=1, confirming that aggressive local training amplifies client drift in temporally skewed social\-event streams\. On PHEME, FedAvg\-CL drops from 73\.8% to 71\.2%, EWC\-FL drops from 78\.6% to 75\.0%, andFedCMMdrops from 86\.3% to 83\.5%\. On CrisisMMD, the absolute decline is larger for the baselines but not by a fixed margin: FedAvg\-CL loses 10\.8%, EWC\-FL loses 9\.3%, andFedCMMloses 5\.45%\. The intermediate points also differ; for example, FedAvg\-CL on PHEME is relatively flat fromE=2E=2toE=3E=3before dropping atE=4E=4, whereas all methods on CrisisMMD show a steeper and more consistent decline\. In particular, the nonparallel degradation shows that FedCMM is not merely an upward\-shifted baseline\. Replay and aggregation filtering reduce the damage caused by longer client\-side optimization, but they do not make large local epochs harmless\. For this reason, the default setting uses one local epoch for the MLLM update, while PPFR’s five fitting epochs apply only to the small replay generator at task boundaries\. The result also suggests that increasing local computation is not a reliable substitute for communication in federated continual MLLM adaptation\.

![Refer to caption](https://arxiv.org/html/2607.12112v1/x4.png)Figure 4:Effect of local epoch numberEEon PHEME and CrisisMMD\.Convergence efficiency\.[Table˜V](https://arxiv.org/html/2607.12112#S5.T5)reports the number of federated rounds needed to first reach 50% and 75% accuracy, and the final accuracy at round 50, underα=0\.5\\alpha=0\.5\. At the 50% milestone,FedCMMconverges in 8 rounds on PHEME and 10 rounds on CrisisMMD, achieving speedups of approximately 2\.25×\\timesover FedAvg\-CL \(18 and 23 rounds\) and 1\.6×\\timesover EWC\-FL \(13 and 15 rounds\)\. At the more demanding 75% target,FedCMMrequires 13 and 19 rounds, while EWC\-FL needs 28 and 33 rounds, roughly twice as many; FedAvg\-CL cannot reach 75% within the 50\-round budget on either benchmark, saturating below that threshold\. At the final checkpoint \(round 50\),FedCMMachieves 86\.3% on PHEME and 84\.4% on CrisisMMD, exceeding EWC\-FL by 7\.7% and 7\.3%, respectively\.

TABLE V:Federated rounds to first reach 50% \(R50↓R\_\{50\}\\downarrow\) and 75% \(R75↓R\_\{75\}\\downarrow\) accuracy, and final accuracy at round 50 \(Final↑\\uparrow\) on PHEME and CrisisMMD\.MethodPHEMECrisisMMDR50↓R\_\{50\}\\downarrowR75↓R\_\{75\}\\downarrowFinal↑\\uparrowR50↓R\_\{50\}\\downarrowR75↓R\_\{75\}\\downarrowFinal↑\\uparrowJoint Training——89\.2——91\.4FedAvg\-CL\[[17](https://arxiv.org/html/2607.12112#bib.bib13)\]18—73\.823—70\.9EWC\-FL\[[10](https://arxiv.org/html/2607.12112#bib.bib16)\]132878\.6153377\.1FedCMM81386\.3101984\.4

### V\-GDiscussion

Superior performance ofFedCMMacross both benchmarks stems from three complementary mechanisms: i\)Modality\-aware regularizationvia MA\-EWC applies module\-specific Fisher penalties to the vision encoder, language backbone, and cross\-modal projector independently, preventing the uniform parameter shrinkage that causes a single\-branch EWC to under\-protect whichever modality updates most aggressively in the current task; ii\)Privacy\-preserving replayvia PPFR restores sample\-space supervision without exposing raw client data, and the empirical gap between PPFR and standard exemplar replay \(GEM\-FL\) confirms that synthetic embeddings preserve enough decision geometry to achieve comparable retention; and iii\)Similarity\-aware aggregationvia TSGA filters out client updates that are directionally misaligned with the consensus, reducing the amplification of task\-ordering drift that ordinary FedAvg cannot suppress\. Nevertheless, several limitations warrant attention: optimalλ\\lambdaandτ\\tauvalues are currently determined by grid search rather than adaptive scheduling; the Fisher diagonal approximation may underestimate parameter importance in cross\-modal projector layers where interference is most nonlinear; and scaling toT≥6T\\\!\\geq\\\!6tasks has not been validated\. Future work will pursue learnable hyperparameter schedules\[[28](https://arxiv.org/html/2607.12112#bib.bib35)\], more expressive Fisher surrogates for low\-rank adapters, and extended evaluation on the seven\-task PHEME→\\rightarrowCrisisMMD combined stream\[[21](https://arxiv.org/html/2607.12112#bib.bib2),[4](https://arxiv.org/html/2607.12112#bib.bib36)\]\.

## VIConclusion

In this paper, we have addressed catastrophic forgetting in the federated fine\-tuning of Multimodal Large Language Models on evolving data streams\. Our proposedFedCMMframework synergistically integrates modality\-aware parameter regularization, raw\-data\-free embedding\-level replay, and similarity\-aware gradient aggregation, enabling MLLMs to learn sequentially without sacrificing prior knowledge\. Extensive experiments show thatFedCMMconsistently achieves the best average accuracy and BWT, while remaining competitive on FWT\. The framework’s robustness to data heterogeneity and asynchronous task ordering further highlights its practical utility for real\-world networked AI deployments\. Future work will focus on improving efficiency and autonomous hyperparameter adaptation, advancing toward evolutive optimization in large\-scale distributed AI systems\.

## References

- \[1\]\(2018\)CrisisMMD: multimodal twitter datasets from natural disasters\.InICWSM,Vol\.12,pp\. 465–467\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p1.1),[§I](https://arxiv.org/html/2607.12112#S1.p2.1),[§V\-A](https://arxiv.org/html/2607.12112#S5.SS1.p1.10)\.
- \[2\]J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millican, M\. Reynolds,et al\.\(2022\)Flamingo: a visual language model for few\-shot learning\.InNeurIPS,Vol\.35,pp\. 23716–23736\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p1.1)\.
- \[3\]Y\. J\. Cho, J\. Wang, T\. Chirvolu, and G\. Joshi\(2023\)Communication\-efficient and model\-heterogeneous personalized federated learning via clustered knowledge transfer\.IEEE J\. Sel\. Top\. Signal Process\.17\(1\),pp\. 40–53\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2022.3231527)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p3.1)\.
- \[4\]J\. Dong, X\. Liu, A\. van den Hengel, J\. He, and C\. Feng\(2024\)Federated continual learning via knowledge fusion: A survey\.IEEE Trans\. Knowl\. Data Eng\.36\(9\),pp\. 4177–4196\.External Links:[Document](https://dx.doi.org/10.1109/TKDE.2024.3353048)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p2.1),[§V\-G](https://arxiv.org/html/2607.12112#S5.SS7.p1.4)\.
- \[5\]R\. Greidi and K\. Cohen\(2024\)Sparse training for federated learning with regularized error correction\.IEEE J\. Sel\. Top\. Signal Process\.\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2024.3463505)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p3.1)\.
- \[6\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InICLR,Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p6.1),[§IV\-A](https://arxiv.org/html/2607.12112#S4.SS1.p1.18),[§V\-B](https://arxiv.org/html/2607.12112#S5.SS2.p1.5)\.
- \[7\]P\. Kairouz, H\. B\. McMahan, B\. Avent, A\. Bellet, M\. Bennis, A\. Bhagoji, K\. Bonawitz,et al\.\(2021\)Advances and open problems in federated learning\.Found\. Trends Mach\. Learn\.14\(1–2\),pp\. 1–210\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p1.1),[§II\-B](https://arxiv.org/html/2607.12112#S2.SS2.p1.1)\.
- \[8\]S\. P\. Karimireddy, S\. Kale, M\. Mohri, S\. Reddi, S\. Stich, and A\. T\. Suresh\(2020\)SCAFFOLD: stochastic controlled averaging for federated learning\.InICML,pp\. 5132–5143\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p4.1),[§II\-B](https://arxiv.org/html/2607.12112#S2.SS2.p1.1)\.
- \[9\]D\. Kiela, H\. Bhooshan, H\. Firooz, and D\. Testuggine\(2020\)The hateful memes challenge: detecting hate speech in multimodal memes\.InNeurIPS,Vol\.33,pp\. 2611–2624\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p1.1),[§I](https://arxiv.org/html/2607.12112#S1.p2.1)\.
- \[10\]J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proc\. Natl\. Acad\. Sci\.114\(13\),pp\. 3521–3526\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p3.1),[§I](https://arxiv.org/html/2607.12112#S1.p6.1),[§II\-A](https://arxiv.org/html/2607.12112#S2.SS1.p1.1),[§II\-C](https://arxiv.org/html/2607.12112#S2.SS3.p1.1),[§IV\-B](https://arxiv.org/html/2607.12112#S4.SS2.p1.1),[§V\-B](https://arxiv.org/html/2607.12112#S5.SS2.p1.5),[TABLE I](https://arxiv.org/html/2607.12112#S5.T1.14.12.19.1),[TABLE IV](https://arxiv.org/html/2607.12112#S5.T4.5.3.5.1),[TABLE V](https://arxiv.org/html/2607.12112#S5.T5.12.6.10.1)\.
- \[11\]E\. Kochkina, M\. Liakata, and A\. Zubiaga\(2018\-08\)All\-in\-one: multi\-task learning for rumour verification\.InCOLING,pp\. 3402–3413\.Cited by:[§V\-A](https://arxiv.org/html/2607.12112#S5.SS1.p1.10)\.
- \[12\]Y\. Lai, L\. Wu, H\. Lin, and J\. Liu\(2025\-10\)Bad\-mfl: a cross\-modality bi\-trigger backdoor attack against multimodal federated learning\.IEEE Internet Things J\.12\(19\),pp\. 39326–39337\.External Links:ISSN 2327\-4662,[Document](https://dx.doi.org/10.1109/JIOT.2025.3589136)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p4.1)\.
- \[13\]T\. Li, A\. K\. Sahu, A\. Talwalkar, and V\. Smith\(2020\)Federated optimization in heterogeneous networks\.MLSys2,pp\. 429–450\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p5.1),[§II\-B](https://arxiv.org/html/2607.12112#S2.SS2.p1.1),[§V\-B](https://arxiv.org/html/2607.12112#S5.SS2.p1.5),[TABLE I](https://arxiv.org/html/2607.12112#S5.T1.14.12.17.1)\.
- \[14\]Z\. Li and D\. Hoiem\(2016\)Learning without forgetting\.InECCV,pp\. 614–629\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p5.1),[§II\-C](https://arxiv.org/html/2607.12112#S2.SS3.p1.1),[§V\-B](https://arxiv.org/html/2607.12112#S5.SS2.p1.5),[TABLE I](https://arxiv.org/html/2607.12112#S5.T1.14.12.18.1)\.
- \[15\]J\. Liu, Y\. Du, K\. Yang, J\. Wu, Y\. Wang, X\. Hu, Z\. Wang, Y\. Liu, P\. Sun, A\. Boukerche, and V\. C\. M\. Leung\(2026\)Edge\-cloud collaborative computing on distributed intelligence and model optimization: a survey\.IEEE Commun\. Surveys Tuts\.28,pp\. 5049–5080\.External Links:ISSN 1553\-877X,[Document](https://dx.doi.org/10.1109/COMST.2026.3669216)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p1.1)\.
- \[16\]D\. Lopez\-Paz and M\. Ranzato\(2017\)Gradient episodic memory for continual learning\.InNeurIPS,Vol\.30\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p3.1),[§II\-A](https://arxiv.org/html/2607.12112#S2.SS1.p1.1),[§V\-B](https://arxiv.org/html/2607.12112#S5.SS2.p1.5),[TABLE I](https://arxiv.org/html/2607.12112#S5.T1.14.12.21.1)\.
- \[17\]B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. A\. y\. Arcas\(2017\)Communication\-efficient learning of deep networks from decentralized data\.InAISTATS,pp\. 1273–1282\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p1.1),[§II\-B](https://arxiv.org/html/2607.12112#S2.SS2.p1.1),[§III](https://arxiv.org/html/2607.12112#S3.p1.16),[§V\-B](https://arxiv.org/html/2607.12112#S5.SS2.p1.5),[TABLE I](https://arxiv.org/html/2607.12112#S5.T1.14.12.16.1),[TABLE II](https://arxiv.org/html/2607.12112#S5.T2.8.6.7.1),[TABLE IV](https://arxiv.org/html/2607.12112#S5.T4.5.3.4.1),[TABLE V](https://arxiv.org/html/2607.12112#S5.T5.12.6.9.1)\.
- \[18\]M\. Mermillod, A\. Bugaiska, and P\. Bonin\(2013\)The stability\-plasticity dilemma: investigating the continuum from catastrophic forgetting to age\-limited learning effects\.Front\. Psychol\.4,pp\. 504\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p3.1),[§II\-A](https://arxiv.org/html/2607.12112#S2.SS1.p1.1)\.
- \[19\]G\. I\. Parisi, R\. Kemker, J\. L\. Part, C\. Kanan, and S\. Wermter\(2019\)Continual lifelong learning with neural networks: a review\.Neural Netw\.113,pp\. 54–71\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p2.1),[§II\-A](https://arxiv.org/html/2607.12112#S2.SS1.p1.1)\.
- \[20\]Z\. Qi, Y\. Tang, L\. Meng, H\. Yu, X\. Li, and X\. Meng\(2025\)Class\-wise balancing data replay for federated class\-incremental learning\.InNeurIPS,Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p5.1),[§II\-C](https://arxiv.org/html/2607.12112#S2.SS3.p1.1)\.
- \[21\]S\. Qian, S\. Zhang, D\. Xue, H\. Zhang, and C\. Xu\(2025\-06\)Learning temporal event knowledge for continual social event classification\.IEEE Trans\. Knowl\. Data Eng\.37\(6\),pp\. 3485–3498\.External Links:ISSN 1558\-2191,[Document](https://dx.doi.org/10.1109/TKDE.2025.3553162)Cited by:[§V\-B](https://arxiv.org/html/2607.12112#S5.SS2.p1.5),[§V\-G](https://arxiv.org/html/2607.12112#S5.SS7.p1.4),[TABLE I](https://arxiv.org/html/2607.12112#S5.T1.14.12.23.1)\.
- \[22\]S\. Rebuffi, A\. Kolesnikov, G\. Sperl, and C\. H\. Lampert\(2017\)ICaRL: incremental classifier and representation learning\.InCVPR,pp\. 2001–2010\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p6.1),[§II\-A](https://arxiv.org/html/2607.12112#S2.SS1.p1.1)\.
- \[23\]S\. J\. Reddi, Z\. Charles, M\. Zaheer, Z\. Garrett, K\. Rush, J\. Konečný, S\. Kumar, and H\. B\. McMahan\(2021\)Adaptive federated optimization\.InICLR,Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p4.1),[§II\-B](https://arxiv.org/html/2607.12112#S2.SS2.p1.1)\.
- \[24\]M\. Ribero, H\. Vikalo, and G\. de Veciana\(2024\)Federated learning at scale: addressing client intermittency and resource constraints\.IEEE J\. Sel\. Top\. Signal Process\.\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2024.3430118)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p1.1)\.
- \[25\]A\. A\. Rusu, N\. C\. Rabinowitz, G\. Desjardins, H\. Soyer, J\. Kirkpatrick, K\. Kavukcuoglu, R\. Pascanu, and R\. Hadsell\(2016\)Progressive neural networks\.InarXiv preprint arXiv:1606\.04671,Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p6.1),[§II\-A](https://arxiv.org/html/2607.12112#S2.SS1.p1.1)\.
- \[26\]G\. Serra and F\. Buettner\(2024\-10\)Federated continual learning goes online: uncertainty\-aware memory management for vision tasks and beyond\.InICLR,Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.12112#S2.SS3.p1.1)\.
- \[27\]H\. Shin, J\. K\. Lee, J\. Kim, and J\. Kim\(2017\)Continual learning with deep generative replay\.InNeurIPS,Vol\.30\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.12112#S2.SS3.p1.1)\.
- \[28\]L\. Wang, X\. Zhang, H\. Su, and J\. Zhu\(2024\)A comprehensive survey of continual learning: theory, method and application\.IEEE Trans\. Pattern Anal\. Mach\. Intell\.46\(8\),pp\. 5057–5076\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2024.3367329)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p2.1),[§V\-G](https://arxiv.org/html/2607.12112#S5.SS7.p1.4)\.
- \[29\]J\. Wu, J\. Liu, Y\. Liu, L\. Wang, Z\. Wang, W\. Chen, Z\. Tian, R\. Yu, and V\. C\. M\. Leung\(2025\-08\)A survey on cloud\-edge\-terminal collaborative intelligence in aiot networks\.External Links:2508\.18803,[Document](https://dx.doi.org/10.48550/arXiv.2508.18803)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p1.1)\.
- \[30\]A\. Wuerkaixi, S\. Cui, J\. Zhang, K\. Yan, B\. Han, G\. Niu, L\. Fang, C\. Zhang, and M\. Sugiyama\(2024\)Accurate forgetting for heterogeneous federated continual learning\.InICLR,Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p5.1),[§II\-C](https://arxiv.org/html/2607.12112#S2.SS3.p1.1),[§V\-B](https://arxiv.org/html/2607.12112#S5.SS2.p1.5),[TABLE I](https://arxiv.org/html/2607.12112#S5.T1.14.12.22.1)\.
- \[31\]T\. Xiao, Y\. Li, Y\. Qi, Y\. Liu, H\. Wang, Y\. Wang, and R\. Li\(2026\-04\)Enhancing privacy in multimodal federated learning with information theory\.NeurIPS38,pp\. 142199–142215\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p5.1)\.
- \[32\]J\. Xu, S\. Wan, Y\. Li, S\. Luo, Z\. Chen, Y\. Shao, Z\. Chen, S\. Huang, and L\. Song\(2024\)Cooperative multi\-model training for personalized federated learning over heterogeneous devices\.IEEE J\. Sel\. Top\. Signal Process\.19,pp\. 195–207\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2024.3497660)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p5.1)\.
- \[33\]R\. Ye, W\. Wang, J\. Chai, D\. Li, Z\. Li, Y\. Xu, Y\. Du, Y\. Wang, and S\. Chen\(2024\)OpenFedLLM: training large language models on decentralized private data via federated learning\.InACM KDD,pp\. 6137–6147\.External Links:[Document](https://dx.doi.org/10.1145/3637528.3671582)Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p6.1)\.
- \[34\]J\. Yoon, W\. Jeong, G\. Lee, E\. Yang, and S\. J\. Hwang\(2021\)Federated continual learning with weighted inter\-client transfer\.InICML,pp\. 12073–12086\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p5.1),[§II\-C](https://arxiv.org/html/2607.12112#S2.SS3.p1.1),[§V\-B](https://arxiv.org/html/2607.12112#S5.SS2.p1.5),[TABLE I](https://arxiv.org/html/2607.12112#S5.T1.14.12.20.1)\.
- \[35\]F\. Zenke, B\. Poole, and S\. Ganguli\(2017\)Continual learning through synaptic intelligence\.InICML,pp\. 3987–3995\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p3.1),[§II\-A](https://arxiv.org/html/2607.12112#S2.SS1.p1.1)\.
- \[36\]Y\. Zhao, M\. Li, L\. Lai, N\. Suda, D\. Civin, and V\. Chandra\(2018\)Federated learning with non\-iid data\.arXiv preprint arXiv:1806\.00582\.Cited by:[§I](https://arxiv.org/html/2607.12112#S1.p3.1),[§II\-B](https://arxiv.org/html/2607.12112#S2.SS2.p1.1)\.