Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing

# Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing
Source: [https://arxiv.org/html/2610.12711](https://arxiv.org/html/2610.12711)
## Byzantine\-Resilient Federated Large Language Model via Hypergraph Message PassingThanks:K\. Li and J\. Calero are with the School of Computer Science and Digital Technologies, Aston University, Birmingham, UK \(E\-mail: kaili@ieee\.org & j\.alcarazcalero@aston\.ac\.uk\)\. H\. Cai, Z\. Liu, Y\. Li, and O\. B\. Akan are with Centre for neXt Communications \(CXC\), Department of Engineering, University of Cambridge, CB3 0FA Cambridge, U\.K\., and O\. B\. Akan is also with the Centre for neXt Communications \(CXC\), Department of Electrical and Electronics Engineering, Koç University, 34450 Istanbul, Türkiye \(e\-mail: \{hc663, zl600, kl596, oba21\}@cam\.ac\.uk & akan@ku\.edu\.tr\)\. W\. Ni is with the School of Engineering, Edith Cowan University, Perth, WA 6027, and the School of Computer Science and Engineering, University of New South Wales, Sydney, NSW 2052, Australia \(E\-mail: wei\.ni@ieee\.org\)\. F\. Dressler is with the Telecommunication Networks Group \(TKN\), School of Electrical Engineering and Computer Science, TU Berlin, Germany \(E\-mail: dressler@ccs\-labs\.org\)\.Thanks:K\. Li and H\. Cai contributed equally to this work\.

Hanlin CaiZihao LiuYichen LiAffiliation:Jose Alcaraz Calero, , Wei Ni, , Falko Dressler, , andAffiliation:Özgür B\. Akan,

###### Abstract

Federated Large Language Models \(FedLLMs\) are promising to deliver privacy\-preserving, collaborative intelligence\. The decentralized training process is vulnerable to Byzantine attacks that corrupt semantic consistency and factual resilience through malicious model updates\. This paper presents a new resilience strategy that prevents semantically corrupted knowledge from propagating during federated aggregation\. A defense problem is formulated to jointly maximize global accuracy and suppress manipulated model updates, enabling trustworthy knowledge aggregation without accessing private local data\. To strengthen the resilience of FedLLMs, we develop a novel Hypergraph Message Passing Graph Autoencoder \(HMP\-GAE\) framework, where low\-rank model updates are represented as nodes in a hypergraph and high\-order semantic relationships among multiple agents are exploited through iterative message passing\. By leveraging historical checkpoints and cross\-agent relational reasoning, HMP\-GAE identifies and suppresses semantically inconsistent updates while preserving knowledge alignment among agents throughout federated training\. HMP\-GAE is implemented using Qwen\-2\.5 and Llama\-3\.2 as pretrained FedLLM backbones and evaluated on AG News and Yahoo\! Answers datasets\. Compared to existing defenses, HMP\-GAE improves global accuracy by1\.211\.21%, reduces final\-model classification semantic entropy by44\.0844\.08%, and lowers perplexity by13\.3413\.34%\. HMP\-GAE outperforms the GAE\-based defense by improving the accuracy1\.391\.39%, reducing the semantic entropy by54\.554\.5%, and lowering perplexity by35\.535\.5%\.

###### Index Terms:

Byzantine Attacks, Federated Large Language Models \(FedLLMs\), Graph Autoencoder \(GAE\), Hypergraph Message Passing, CyberEdge

## IIntroduction

Large language models \(LLMs\) are emerging as a transformative paradigm for intelligent, autonomous operations in CyberEdge networks\[[1](https://arxiv.org/html/2610.12711#bib.bib38),[2](https://arxiv.org/html/2610.12711#bib.bib36)\], where distributed devices, from IoT sensors to connected vehicles and industrial robots, collaborate securely and efficiently without centralized control\. By integrating federated learning \(FL\) with LLMs, edge nodes can perform local reasoning, decision\-making, and adaptation based on their contextual data while sharing high\-level knowledge or model updates rather than raw information\[[3](https://arxiv.org/html/2610.12711#bib.bib16),[4](https://arxiv.org/html/2610.12711#bib.bib37),[5](https://arxiv.org/html/2610.12711#bib.bib39)\]\. This enables a new generation of self\-organizing, privacy\-preserving, and resilient CyberEdge networks that can support complex, real\-time applications, e\.g\., autonomous mobility, digital twin synchronization, smart healthcare, and disaster response\. Federated LLMs \(FedLLMs\) act as cognitive coordinators, learning human\-aligned behaviors, negotiating across agents, and dynamically optimizing communication, computation, and sensing processes under constrained resources\[[6](https://arxiv.org/html/2610.12711#bib.bib33),[7](https://arxiv.org/html/2610.12711#bib.bib35),[8](https://arxiv.org/html/2610.12711#bib.bib34),[9](https://arxiv.org/html/2610.12711#bib.bib28)\]\. This synergy enhances situational awareness, interoperability, and trust across heterogeneous edge environments, paving the way to autonomous, explainable, and resilient artificial intelligence \(AI\) infrastructures at the network edge\.

In FedLLMs, each edge device or agent maintains a local instance of an LLM that learns from its own private data, such as text or sensory streams, arising from its specific environment and task context\. Instead of transmitting raw data to a central server, participants share only model updates or gradients through secure aggregation, collaboratively refining a global foundation model\. The nature of FedLLMs empowers each agent to integrate and reason across diverse data features, dynamically adapt its learning strategy to evolving local contexts, and coordinate with other agents based on shared semantics, trust levels, and task interdependencies\[[10](https://arxiv.org/html/2610.12711#bib.bib45),[11](https://arxiv.org/html/2610.12711#bib.bib31),[12](https://arxiv.org/html/2610.12711#bib.bib32),[13](https://arxiv.org/html/2610.12711#bib.bib29)\]\. Through iterative aggregation and local fine\-tuning, FedLLMs collectively evolve into a distributed intelligence capable of robust language understanding, cross\-modal reasoning, and context\-aware decision\-making in heterogeneous and privacy\-sensitive CyberEdge environments\.

Within this collaborative framework, FedLLMs are particularly vulnerable to Byzantine attacks, where adversaries exploit the open\-ended generative capacity of LLMs and their internal representation spaces to propagate false or biased knowledge across agents\[[14](https://arxiv.org/html/2610.12711#bib.bib17),[15](https://arxiv.org/html/2610.12711#bib.bib19),[16](https://arxiv.org/html/2610.12711#bib.bib46)\]\.

As depicted in Fig\.[1](https://arxiv.org/html/2610.12711#S1.F1), a malicious client or compromised edge agent may inject subtle cross\-modal perturbations, such as corrupted textual prompts, mislabeled visual samples, or biased sensory data, into its local training process, steering the local model toward distorted semantic associations or fabricated knowledge\[[17](https://arxiv.org/html/2610.12711#bib.bib26)\]\. When these adversarial updates are aggregated, the global model’s latent graph and contextual alignments be corrupted, leading to systemic manipulations that produce confident yet false textual descriptions, visual interpretations, or sensor\-based inferences\. Over multiple communication rounds, these manipulations can cascade through the network, undermining the collective intelligence of FedLLMs\. For instance, a malicious agent in a vehicular CyberEdge network might cause an LLM to create nonexistent obstacles from camera feeds, while in healthcare or industrial IoT systems, fabricated insights could misguide diagnostics or trigger unsafe automated actions\.

Moreover, the distributed and privacy\-preserving nature of FedLLMs amplifies the impact of Byzantine attacks, since each agent trains locally on private data, and centralized verification or direct inspection of training samples is infeasible\. A single compromised agent can influence others through conversational interactions, aggregation of model updates, or semantic alignment mechanisms, gradually inducing large\-scale semantic drift in the collective model\. Such drift erodes the interpretability, factual consistency, and ethical resilience of the FedLLMs\.

![Refer to caption](https://arxiv.org/html/2610.12711v1/FedLLMs.png)Fig\. 1:In FedLLMs, users train their local LLMs and produce local model updates\.This paper presents a new Byzantine\-resilient strategy to mitigate Byzantine attacks in FedLLMs that distort semantic consistency and induce factual misalignment in LLMs\. The strategy introduces a self\-healing mechanism within the federated aggregation process to recognize and neutralize adversarial model updates before they propagate semantic corruption across FedLLMs\. A problem is cast to maximize task accuracy of the LLMs while simultaneously identifying and suppressing manipulated updates\. This filters for trustworthy and semantically coherent knowledge, thereby improving the learned representations\.

To adaptively enhance the resilience of FedLLMs against latent Byzantine\-inducing perturbations, we design theHypergraph Message Passing Graph Autoencoder \(HMP\-GAE\) framework\. In HMP\-GAE, at the server each agent’s model update that is fine\-tuned by low\-rank adaptation \(LoRA\) is represented as a node within a hypergraph, where hyperedges capture high\-order semantic and contextual relationships among multiple agent updates, as opposed to simple pairwise correlations\. The server performs iterative message passing across these hyperedges to refine the embeddings of benign updates, allowing collective reasoning that effectively identifies and suppresses corrupted or semantically inconsistent contributions\. By leveraging hypergraph structures, HMP\-GAE distills reliable relational knowledge from trustworthy agents and historical checkpoints, ensuring robust aggregation that maintains factual coherence and mitigates adversarial manipulation\.

The contributions of this paper are listed as follows:

- •This paper proposes the first graph neural network\-based strategy for mitigating Byzantine attacks in FedLLMs, which enhances the resilience of FedLLMs against subtle adversarial perturbations\. The strategy introduces a defense perspective that focuses on preserving semantic integrity and factual consistency across distributed FL\. We formulate a defense problem at benign users to maximize global task accuracy while filtering out malicious, semantically corrupted contributions during aggregation\.
- •We design the HMP\-GAE framework to enhance FedLLMs against Byzantine attacks through structured relational reasoning and representation alignment\. By modeling each agent’s update as a node in a hypergraph and capturing high\-order semantic relationships via message passing, HMP\-GAE identifies and suppresses semantically inconsistent contributions\. This hypergraph\-based aggregation mechanism strengthens cross\-agent knowledge coherence, preserves reasoning integrity, and mitigates adversarial manipulation\.
- •Extensive experiments are conducted under diverse adversarial scenarios to evaluate the HMP\-GAE framework\. We assess key metrics, including prediction accuracy, semantic entropy, and perplexity\. HMP\-GAE achieves superior accuracy and resilience while maintaining low resource overhead, validating its practicality as an effective and scalable Byzantine\-resilient defense for FedLLMs under malicious model updates\. The source code of the proposed HMP\-GAE is released on GitHub:[https://github\.com/GuangLun2000/HMP\-GAE](https://github.com/GuangLun2000/HMP-GAE)\.

The remainder of this paper is structured as follows\. Section[II](https://arxiv.org/html/2610.12711#S2)reviews the literature on Byzantine\-resilient mitigation, assessment, and detection in FL and LLMs\. Section[III](https://arxiv.org/html/2610.12711#S3)investigates the state\-of\-the\-art training architecture of FedLLMs as well as the threat model of Byzantine attacks\. The proposed HMP\-GAE for mitigating Byzantine attacks is described in Section[IV](https://arxiv.org/html/2610.12711#S4)\. Section[V](https://arxiv.org/html/2610.12711#S5)presents the experimental implementation in PyTorch, and resilience performance analysis\. Section[VI](https://arxiv.org/html/2610.12711#S6)concludes this paper\.

## IIRelated Work

This section reviews the existing studies on Byzantine attacks mitigation and detection in FL\. Our contributions over the existing Byzantine\-resilient models are presented\.

### II\-AByzantine\-resilient Strategies

The authors in\[[18](https://arxiv.org/html/2610.12711#bib.bib24)\]developed a communication\-efficient and Byzantine\-tolerant FL in CyberEdge networks\. It combined sign\-refined momentum updates with error compensation to reduce communication costs and stabilize convergence\. Edge servers used cosine similarity to assess trust and suppress unreliable local updates\. The cloud server combined real and dummy data projections through projection consensus for global aggregation\. Layer\-wise cosine similarity was used in\[[19](https://arxiv.org/html/2610.12711#bib.bib20)\]to detect malicious gradients against a gradient median benchmark\. A proactive defense was formulated to constrain divergence among benign gradients, while trust scores reduce misjudgments under non\-IID data\.

In\[[20](https://arxiv.org/html/2610.12711#bib.bib7)\], a Byzantine\-resilient FL was studied for improving global model consistency\. An asynchronous common subset protocol was presented, where benign users can aggregate the model updates despite conflicting messages from malicious users\. Dual\-domain trust scoring was used to evaluate consensus behavior, similarity among model updates, and stability across rounds to limit malicious influence\. Shen et al\.\[[21](https://arxiv.org/html/2610.12711#bib.bib22)\]presented a Byzantine\-resilient FL, which can distinguish malicious model updates through discrepancies in local models’ feature representations\. A reference model trained on public data was utilized to quantify these discrepancies as feature fingerprints\. These fingerprints were clustered to filter out malicious models before aggregation\.

Two FL schemes were developed in\[[22](https://arxiv.org/html/2610.12711#bib.bib18)\]to enhance resilience against Byzantine users while preventing eavesdroppers from learning users’ private data\. Both used a small representative dataset at the server and a discriminator function to suppress malicious contributions\. Lagrange coded computing was combined with re\-randomization to protect privacy, while communication costs were reduced through a trusted third party\. Moreover, a sparsity\-guided refinement scheme could be added to mitigate Byzantine attacks\[[23](https://arxiv.org/html/2610.12711#bib.bib10)\]\. Sparsity was utilized to derive significance distribution vectors that expose subtle malicious changes in client model updates\. These vectors are clustered to compute weighted similarity scores for group resilience\. The resulting groups guide further refinement of raw updates before robust aggregation\.

Bray\-Curtis dissimilarity was introduced in\[[24](https://arxiv.org/html/2610.12711#bib.bib12)\]to measure element\-wise differences among users’ gradients\. Their method can reduce the misclassification of benign gradients against Byzantine attacks under non\-IID data distributions\. A lattice\-based fully homomorphic encryption scheme was used to protect local gradients while enabling secure aggregation and Byzantine user detection\. The authors in\[[25](https://arxiv.org/html/2610.12711#bib.bib11)\]formulated a Byzantine\-resilient FL scheme combining homomorphic encryption with gradient dimension compression\. The Johnson\-Lindenstrauss transformation was applied to reduce gradient dimensions while preserving geometric relationships needed for Byzantine defense\.

Existing Byzantine\-resilient aggregation rules were adapted in\[[26](https://arxiv.org/html/2610.12711#bib.bib23)\]\. A shared, updated consensus dataset was introduced to provide a common benchmark for applying these rules and reaching agreement\. A dual\-layer privacy shielding strategy was designed to protect local models while preserving global model accuracy\. A Byzantine\-resilient FL framework was developed in\[[27](https://arxiv.org/html/2610.12711#bib.bib21)\], combining client\-side and server\-side defenses\. Clients used a model\-level penalty loss to limit divergence from benign models and reduce correlation with malicious model updates, which can help prevent benign users from being misclassified\. The server combined manifold\-based analysis with a maximum\-clique strategy to strengthen Byzantine detection\.

Wang et al\.\[[28](https://arxiv.org/html/2610.12711#bib.bib15)\]presented a Byzantine\-resilient scheme to defend FL against poisoning attacks while protecting user privacy\. A two\-server architecture used secure two\-party computation to analyze local model gradients\. A security protocol was designed to normalize users’ model gradients, compute their cosine similarities, and identify malicious model updates\. In\[[29](https://arxiv.org/html/2610.12711#bib.bib14)\], a two\-stage defense framework was presented for Byzantine\-resilient FL\. Malicious users can be detected through model similarity\. A validation mechanism was then used to estimate similarity among users’ local data to refine detection decisions\. A weight recovery mechanism was also applied to adjust the aggregation weights of users identified as Byzantine\.

In\[[30](https://arxiv.org/html/2610.12711#bib.bib27)\], dimensionality reduction was adopted to accelerate the private computation of complex filtering rules in Byzantine\-resilient FL\. An adaptive tuning strategy was formulated to limit the influence of malicious model updates that evade filtering in the reduced space\. Moreover, the authors in\[[31](https://arxiv.org/html/2610.12711#bib.bib9)\]introduced a correction term during local training to reduce model inconsistency\. A secure protocol deployed across two servers enabled Byzantine\-resilient aggregation while protecting user privacy, which was expected to limit the privacy computation burden on resource\-constrained users\.

Herath et al\.\[[32](https://arxiv.org/html/2610.12711#bib.bib8)\]developed a dual\-server protocol to protect local model updates without encryption or key exchange\. A group\-wise filtering mechanism used deviation scores and client credits to isolate Byzantine users, where a reward\-penalty system can promote fair participation throughout training\. To address heavy\-tailed gradient noise, the authors in\[[33](https://arxiv.org/html/2610.12711#bib.bib2)\]combined distributed stochastic heavy\-ball optimization with update normalization and gradient clipping\. Nearest\-neighbor mixing through random projections was used to isolate the impact of Byzantine users\.

An internal auditor was built in\[[34](https://arxiv.org/html/2610.12711#bib.bib13)\]to examine gradient similarity and distribution to distinguish benign model updates from malicious ones\. It combined a Gaussian mixture model with Mahalanobis distance for Byzantine\-tolerant aggregation\. A three\-server computation was developed to perform Byzantine\-resilient aggregation\[[35](https://arxiv.org/html/2610.12711#bib.bib25)\], where each server held two of the three shares for privacy protection\. A top\-k protocol was invoked to measure the cosine distance and compute the scores to select benign local models\.

### II\-BOur Contributions

Existing Byzantine\-resilient techniques primarily address Byzantine attacks in FL\. Their defenses employ robust aggregation, model update similarity, trust evaluation, and secure computation to identify malicious contributions\. However, Byzantine resilience in FedLLMs has not been specifically addressed\. Our work aims to investigate Byzantine\-resilient FedLLMs, focusing on how malicious updates disrupt semantic consistency and knowledge alignment during federated aggregation\.

Moreover, our proposed HMP\-GAE framework for Byzantine\-resilient FedLLMs enables each user’s LoRA update to form a hypergraph node, while hyperedges capture semantic relationships among multiple users\. HMP\-GAE combines historical checkpoints with iterative hypergraph message passing to learn shared latent representations of these updates\. HMP\-GAE captures high\-order relational inconsistencies that individual model update analysis may overlook\. Therefore, the representations are leveraged to identify and suppress semantically inconsistent updates during aggregation, which enables Byzantine\-resilient knowledge integration without accessing private local data\.

## IIIFormulation of FedLLMs and Threat Model

In this section, we first formulate collaborative adaptation of a shared pretrained LLM and then describe Byzantine manipulation of the aggregated parameters\.

### III\-AFine\-tuning of FedLLMs

Consider FedLLMs withK≥2K\\geq 2agents participating in each ofRRcommunication rounds, wherer=0,1,…,R−1r=0,1,\\ldots,R\-1\. Letθ∈ℝdθ\\theta\\in\\mathbb\{R\}^\{d\_\{\\theta\}\}denote the frozen pretrained backbone parameters that are not updated during federated fine\-tuning\. The benign and Byzantine agents form disjoint setsℬ\\mathcal\{B\}and𝒜\\mathcal\{A\}, respectively, withℬ∪𝒜=\{1,…,K\}\\mathcal\{B\}\\cup\\mathcal\{A\}=\\\{1,\\ldots,K\\\}\. Agentkkholds a private dataset\{\(xk,n,yk,n\)\}n=1Nk\\\{\(x\_\{k,n\},y\_\{k,n\}\)\\\}\_\{n=1\}^\{N\_\{k\}\}, whereNk\>0N\_\{k\}\>0is the number of data samples at the agent,xk,nx\_\{k,n\}is the input text, andyk,n=\(yk,n1,…,yk,nTk,n\)y\_\{k,n\}=\(y\_\{k,n\}^\{1\},\\ldots,y\_\{k,n\}^\{T\_\{k,n\}\}\)is the target token sequence of lengthTk,nT\_\{k,n\}\.

To condition the LLM, the input encoderfϕf\_\{\\phi\}and the context projection modulegψg\_\{\\psi\}produce\[[36](https://arxiv.org/html/2610.12711#bib.bib1)\]

ck,n=gψ​\(fϕ​\(xk,n\)\),\\displaystyle c\_\{k,n\}=g\_\{\\psi\}\(f\_\{\\phi\}\(x\_\{k,n\}\)\),\(1\)where the trainable parametersϕ\\phiandψ\\psiare separate from the frozen backboneθ\\theta\.

We adopt LoRA for the backbone\. Let𝒮\\mathcal\{S\}index the adapted projections\. For eachℓ∈𝒮\\ell\\in\\mathcal\{S\}, the pre\-trained matrixWℓ∈ℝaℓ×bℓW\_\{\\ell\}\\in\\mathbb\{R\}^\{a\_\{\\ell\}\\times b\_\{\\ell\}\}remains frozen, and its updateΔ​Wℓ\\Delta W\_\{\\ell\}is parameterized as

Δ​Wℓ\\displaystyle\\Delta W\_\{\\ell\}=αℓqℓ​Bℓ​Aℓ,\\displaystyle=\\frac\{\\alpha\_\{\\ell\}\}\{q\_\{\\ell\}\}B\_\{\\ell\}A\_\{\\ell\},\(2\)Aℓ\\displaystyle A\_\{\\ell\}∈ℝqℓ×bℓ,Bℓ∈ℝaℓ×qℓ\.\\displaystyle\\in\\mathbb\{R\}^\{q\_\{\\ell\}\\times b\_\{\\ell\}\},\\;\\;\\;B\_\{\\ell\}\\in\\mathbb\{R\}^\{a\_\{\\ell\}\\times q\_\{\\ell\}\}\.whereaℓa\_\{\\ell\}andbℓb\_\{\\ell\}denote the output and input dimensions of projectionℓ\\ell, respectively;qℓ≪min⁡\(aℓ,bℓ\)q\_\{\\ell\}\\ll\\min\(a\_\{\\ell\},b\_\{\\ell\}\)is the adaptation rank; andαℓ\\alpha\_\{\\ell\}is a scaling parameter\. The trainable factorsAℓA\_\{\\ell\}andBℓB\_\{\\ell\}map the input to a rank\-qℓq\_\{\\ell\}space and then to the output space, respectively\.

Letλ=\{Aℓ,Bℓ\}ℓ∈𝒮\\lambda=\\\{A\_\{\\ell\},B\_\{\\ell\}\\\}\_\{\\ell\\in\\mathcal\{S\}\}collect the LoRA factors\. The backbone updateΔ​θ​\(λ\)\\Delta\\theta\(\\lambda\)places the vectorized matricesΔ​Wℓ\\Delta W\_\{\\ell\}at their corresponding parameter locations and is zero elsewhere\. Thus, the backbone isθ\+Δ​θ​\(λ\)\\theta\+\\Delta\\theta\(\\lambda\), and the trainable parameter collection is given by

ω=\(λ,ϕ,ψ\),\\displaystyle\\omega=\(\\lambda,\\phi,\\psi\),\(3\)
The model’s predictive distribution can be given aspθ,ωp\_\{\\theta,\\omega\}, and autoregressive factorization is

pθ,ω​\(yk,n∣xk,n\)\\displaystyle p\_\{\\theta,\\omega\}\(y\_\{k,n\}\\\!\\mid\\\!x\_\{k,n\}\)\\\!=∏j=1Tk,np⁡\(yk,nj∣yk,n<j,ck,n;θ\+Δ​θ​\(λ\)\),\\displaystyle=\\\!\\prod\_\{j=1\}^\{T\_\{k,n\}\}p\(y\_\{k,n\}^\{j\}\\\!\\mid\\\!y\_\{k,n\}^\{<j\},c\_\{k,n\};\\theta\\\!\+\\\!\\Delta\\theta\(\\lambda\)\),\(4\)wherejjindexes target tokens, andyk,n<jy\_\{k,n\}^\{<j\}contains the preceding tokens\. The local objective averages sequence negative log\-likelihoods across samples following

ℒk\(ω\)=−1Nk∑n=1Nk∑j=1Tk,nlogp\(yk,nj∣yk,n<j,ck,n;θ\+Δθ\(λ\)\)\.\\displaystyle\\mathcal\{L\}\_\{k\}\(\\omega\)=\-\\frac\{1\}\{N\_\{k\}\}\\sum\_\{n=1\}^\{N\_\{k\}\}\\sum\_\{j=1\}^\{T\_\{k,n\}\}\\log p\(y\_\{k,n\}^\{j\}\\mid y\_\{k,n\}^\{<j\},c\_\{k,n\};\\theta\+\\Delta\\theta\(\\lambda\)\)\.\(5\)Thus, the FedLLMs’ objective can be formulated as

minω⁡F⁡\(ω\)\\displaystyle\\min\_\{\\omega\}F\(\\omega\)=∑k=1Kpk​ℒk​\(ω\)​with​pk=Nk∑i=1KNi,\\displaystyle=\\sum\_\{k=1\}^\{K\}p\_\{k\}\\mathcal\{L\}\_\{k\}\(\\omega\)\\;\\text\{ with \}\\;p\_\{k\}=\\frac\{N\_\{k\}\}\{\\sum\_\{i=1\}^\{K\}N\_\{i\}\},\(6\)wherepkp\_\{k\}describes the intended collaborative objective\.

At communication roundrr, the server broadcastsωr\\omega^\{r\}\. Each benign agentk∈ℬk\\in\\mathcal\{B\}initializesωkr,0=ωr\\omega\_\{k\}^\{r,0\}=\\omega^\{r\}and performsEElocal stochastic gradient descent steps as

ωkr,e\+1\\displaystyle\\omega\_\{k\}^\{r,e\+1\}=ωkr,e−ηr,e​∇ωℒ^k​\(ωkr,e,ℳkr,e\),\\displaystyle=\\omega\_\{k\}^\{r,e\}\-\\eta\_\{r,e\}\\nabla\_\{\\omega\}\\widehat\{\\mathcal\{L\}\}\_\{k\}\(\\omega\_\{k\}^\{r,e\};\\mathcal\{M\}\_\{k\}^\{r,e\}\),\(7\)e\\displaystyle e=0,1,…,E−1,\\displaystyle=0,1,\\ldots,E\-1,whereηr,e\\eta\_\{r,e\}is the learning rate,ℳkr,e\\mathcal\{M\}\_\{k\}^\{r,e\}is a sampled minibatch, andℒ^k\\widehat\{\\mathcal\{L\}\}\_\{k\}is its loss estimate\. The model update from agentkkis denoted by

ω^kr=\(\{A^k,ℓr,B^k,ℓr\}ℓ∈𝒮,ϕ^kr,ψ^kr\)\.\\displaystyle\\widehat\{\\omega\}\_\{k\}^\{r\}=\(\\\{\\widehat\{A\}\_\{k,\\ell\}^\{r\},\\widehat\{B\}\_\{k,\\ell\}^\{r\}\\\}\_\{\\ell\\in\\mathcal\{S\}\},\\widehat\{\\phi\}\_\{k\}^\{r\},\\widehat\{\\psi\}\_\{k\}^\{r\}\)\.\(8\)We haveω^kr=ωkr,E\\widehat\{\\omega\}\_\{k\}^\{r\}=\\omega\_\{k\}^\{r,E\},∀k∈ℬ\\forall k\\in\\mathcal\{B\}\.

Givenℓ∈𝒮\\ell\\in\\mathcal\{S\}, the server assigns weightsp~kr≥0\\tilde\{p\}\_\{k\}^\{r\}\\geq 0that have to satisfy∑k=1Kp~kr=1\\sum\_\{k=1\}^\{K\}\\tilde\{p\}\_\{k\}^\{r\}=1\. The additional trainable modules adopt the same weights

ϕr\+1\\displaystyle\\phi^\{r\+1\}=∑k=1Kp~kr​ϕ^kr,ψr\+1=∑k=1Kp~kr​ψ^kr\.\\displaystyle=\\sum\_\{k=1\}^\{K\}\\tilde\{p\}\_\{k\}^\{r\}\\widehat\{\\phi\}\_\{k\}^\{r\},\\;\\;\\;\\psi^\{r\+1\}=\\sum\_\{k=1\}^\{K\}\\tilde\{p\}\_\{k\}^\{r\}\\widehat\{\\psi\}\_\{k\}^\{r\}\.\(9\)Therefore, the global backbone is given as

θr\+1=θ\+Δ​θ​\(λr\+1\)\.\\displaystyle\\theta^\{r\+1\}=\\theta\+\\Delta\\theta\(\\lambda^\{r\+1\}\)\.\(10\)

### III\-BThreat Model of Byzantine Attacks

Compromised agents, i\.e\., agenta∈𝒜a\\in\\mathcal\{A\}, can upload arbitrary finite parameters, subject to the proximity constraint specified below\. They receive the broadcast global models but cannot access the benign agents’ private datasets\. Let𝒬\\mathcal\{Q\}denote an evaluation distribution over input\-target pairs\(x,y\)\(x,y\)\. The averages attack success rate over𝒬\\mathcal\{Q\}can be defined as

ℛadv​\(ω\)\\displaystyle\\mathcal\{R\}^\{\\mathrm\{adv\}\}\(\\omega\)=𝔼\(x,y\)∼𝒬\[hadv\(pθ,ω\(⋅∣x\),y\)\],\\displaystyle=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{Q\}\}\[h\_\{\\mathrm\{adv\}\}\(p\_\{\\theta,\\omega\}\(\\cdot\\mid x\),y\)\],\(11\)
To reduce detectability, each malicious agenta∈𝒜a\\in\\mathcal\{A\}constrains the deviation of its LoRA weights from those of the current broadcast global model\. At communication roundrr, letB^a,ℓr\\widehat\{B\}\_\{a,\\ell\}^\{r\}andA^a,ℓr\\widehat\{A\}\_\{a,\\ell\}^\{r\},∀ℓ∈𝒮\\forall\\ell\\in\\mathcal\{S\}denote the LoRA factors of malicious agenta∈𝒜a\\in\\mathcal\{A\}\. The matricesBℓrB\_\{\\ell\}^\{r\}andAℓrA\_\{\\ell\}^\{r\}are the global LoRA factors parameterizing the weight update to the frozen pretrained layerℓ\\ellat roundrr\. According to\[[37](https://arxiv.org/html/2610.12711#bib.bib6),[38](https://arxiv.org/html/2610.12711#bib.bib5)\], the weight change at adapted layerℓ\\ellis given by

Ua,ℓr\\displaystyle U\_\{a,\\ell\}^\{r\}=αℓqℓ​\(B^a,ℓr​A^a,ℓr−Bℓr​Aℓr\)\.\\displaystyle=\\frac\{\\alpha\_\{\\ell\}\}\{q\_\{\\ell\}\}\(\\widehat\{B\}\_\{a,\\ell\}^\{r\}\\widehat\{A\}\_\{a,\\ell\}^\{r\}\-B\_\{\\ell\}^\{r\}A\_\{\\ell\}^\{r\}\)\.\(12\)
At communication roundrr, the malicious agents seek to maximize the risk of the resulting aggregated model as

max\{ω^ar\}a∈𝒜⁡ℛadv​\(ωr\+1\)\\displaystyle\\max\_\{\\\{\\widehat\{\\omega\}\_\{a\}^\{r\}\\\}\_\{a\\in\\mathcal\{A\}\}\}\\mathcal\{R\}^\{\\mathrm\{adv\}\}\(\\omega^\{r\+1\}\)s\.t\.​ωr\+1=∑k=1Kp~kr​ω^kr,\\displaystyle\\text\{s\.t\.\}\\;\\;\\;\\omega^\{r\+1\}=\\sum\_\{k=1\}^\{K\}\\tilde\{p\}\_\{k\}^\{r\}\\widehat\{\\omega\}\_\{k\}^\{r\},\(13\)ω^kr=ωkr,E,\\displaystyle\\qquad\\widehat\{\\omega\}\_\{k\}^\{r\}=\\omega\_\{k\}^\{r,E\},\(14\)∑ℓ‖Ua,ℓr‖F2≤ε2,∀a∈𝒜\.\\displaystyle\\qquad\\sum\_\{\\ell\}\\\|U\_\{a,\\ell\}^\{r\}\\\|\_\{F\}^\{2\}\\leq\\varepsilon^\{2\},\\;\\forall a\\in\\mathcal\{A\}\.\(15\)Constraint \([15](https://arxiv.org/html/2610.12711#S3.E15)\) runs over LoRA\-adapted layers\.∥⋅∥F\\\|\\cdot\\\|\_\{F\}denotes the Frobenius norm\.ε\>0\\varepsilon\>0bounds the total deviation in the LoRA weights\.

![Refer to caption](https://arxiv.org/html/2610.12711v1/system.png)Fig\. 2:The architecture of the proposed HMP\-GAE framework, which is applied at the server\. By modeling each agent’s update as a node in a hypergraph and capturing high\-order semantic relationships via message passing, HMP is developed with GAE to identify and suppress semantically inconsistent or malicious contributions\.

## IVHMP\-GAE for Byzantine\-Resilient FedLLMs

In this section, HMP\-GAE is designed to assess aggregated model updates before aggregation, which combines HMP with GAE to capture relationships among multiple agents and identify potentially inconsistent contributions\. The server uses aggregated parameters and historical checkpoints without accessing private local training data\.

### IV\-AArchitecture of the Proposed HMP\-GAE

As shown in Fig\.[2](https://arxiv.org/html/2610.12711#S3.F2), the server processesKKmodel updates through the graph embedding layer, where the benign and Byzantine identities are unknown to the defense\. LetΘ\\Thetacollect the trainable parameters of HMP\-GAE, including its feature embedding network, GAE encoders, and decoders\.

#### IV\-A1Update Representation

The server can reconstruct the change in weights relative to the broadcast global model

Uk,ℓr=αℓqℓ​\(B^k,ℓr​A^k,ℓr−Bℓr​Aℓr\),\\displaystyle U\_\{k,\\ell\}^\{r\}=\\frac\{\\alpha\_\{\\ell\}\}\{q\_\{\\ell\}\}\(\\widehat\{B\}\_\{k,\\ell\}^\{r\}\\widehat\{A\}\_\{k,\\ell\}^\{r\}\-B\_\{\\ell\}^\{r\}A\_\{\\ell\}^\{r\}\),\(16\)whereA^k,ℓr\\widehat\{A\}\_\{k,\\ell\}^\{r\}andB^k,ℓr\\widehat\{B\}\_\{k,\\ell\}^\{r\}are the LoRA factor matrices uploaded by agentkk\.

Furthermore, each reconstructed matrixUk,ℓrU\_\{k,\\ell\}^\{r\}measures the difference between the additive weight updates represented by agentkk’s LoRA factors and the global LoRA factors\. The layer\-wise differences can be concatenated with the changes in the input encoder parametersϕ\\phiand context projection parametersψ\\psito form a single descriptor of the agent’s model update\. Letvec⁡\(⋅\)\\operatorname\{vec\}\(\\cdot\)denote column\-wise vectorization, and the LLM descriptor can be formulated as

ukr=concat⁡\(CLOSE\\displaystyle u\_\{k\}^\{r\}=\\operatorname\{concat\}\\hbox\{$\\left\(\\vbox to0\.0pt\{\}\\right\.$\}\{vec⁡\(Uk,ℓr\)\}ℓ∈𝒮,\\displaystyle\\\{\\operatorname\{vec\}\(U\_\{k,\\ell\}^\{r\}\)\\\}\_\{\\ell\\in\\mathcal\{S\}\},vec⁡\(ϕ^kr−ϕr\),vec⁡\(ψ^kr−ψr\)​\)∈ℝD,\\displaystyle\\operatorname\{vec\}\(\\widehat\{\\phi\}\_\{k\}^\{r\}\-\\phi^\{r\}\),\\operatorname\{vec\}\(\\widehat\{\\psi\}\_\{k\}^\{r\}\-\\psi^\{r\}\)\\hbox\{$\\left\)\\vbox to0\.0pt\{\}\\right\.$\}\\in\\mathbb\{R\}^\{D\},\(17\)whereDDrepresents the descriptor’s dimension\.

For a uniform architectural description, all node and hyperedge embeddings have widthdd\. A feature network as a trainable mapping can combine the model\-update descriptorukru\_\{k\}^\{r\}with the historical embeddinghkr−1∈ℝdh\_\{k\}^\{r\-1\}\\in\\mathbb\{R\}^\{d\},hk−1=𝟎h\_\{k\}^\{\-1\}=\\mathbf\{0\}, and map them into the shareddd\-dimensional feature space\. Thus, we have

Ωkr=fenc,ξ\(ukr,hkr−1\)∈ℝd,k=1,…,K,\\displaystyle\\Omega\_\{k\}^\{r\}=f\_\{\\mathrm\{enc\},\\xi\}\(u\_\{k\}^\{r\},h\_\{k\}^\{r\-1\}\)\\in\\mathbb\{R\}^\{d\},\\;\\;k=1,\\ldots,K,\(18\)whereξ⊆Θ\\xi\\subseteq\\Thetadenotes the parameters\. Then, stacking the node features gives

Xr=\[Ω1r,…,ΩKr\]⊤∈ℝK×d\.\\displaystyle X^\{r\}=\[\\Omega\_\{1\}^\{r\},\\ldots,\\Omega\_\{K\}^\{r\}\]^\{\\top\}\\in\\mathbb\{R\}^\{K\\times d\}\.\(19\)

#### IV\-A2Hypergraph Construction

Since a hyperedge can jointly connect multiple agents with related update representations\. This allows HMP\-GAE to assess each update within a group of agents and capture relational patterns that isolated or pairwise comparisons may overlook\. Moreover, an agent can belong to multiple overlapping hyperedges, allowing its update to be evaluated across different groups\.

At communication roundrr, the hypergraphℋr=\(𝒱r,ℰr\)\\mathcal\{H\}^\{r\}=\(\\mathcal\{V\}^\{r\},\\mathcal\{E\}^\{r\}\)contains one nodevkv\_\{k\}per agent\.

Let𝒩sr​\(k\)\\mathcal\{N\}\_\{s\}^\{r\}\(k\)contain thessnearest nodes to nodevkv\_\{k\}under Euclidean distance or Cosine similarity between the initial node features, where1≤s<K1\\leq s<Kand ties are resolved by agent index\. GivenKKhyperedges, each agent defines a hyperedge

ekr=\{vk\}∪\{vi:i∈𝒩sr​\(k\)\}\.\\displaystyle e\_\{k\}^\{r\}=\\\{v\_\{k\}\\\}\\cup\\\{v\_\{i\}:i\\in\\mathcal\{N\}\_\{s\}^\{r\}\(k\)\\\}\.\(20\)The incidence matrix isHr∈\{0,1\}K×KH^\{r\}\\in\\\{0,1\\\}^\{K\\times K\}with the\(i,k\)\(i,k\)\-th element given by

Hi​kr=\{1,if​vi∈ekr;0,otherwise\.\\displaystyle H\_\{ik\}^\{r\}=\\begin\{cases\}1,&\\text\{if \}v\_\{i\}\\in e\_\{k\}^\{r\};\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(21\)
For simplification, the round superscriptris omitted within the following message\-passing and reconstruction equations\. The degree matrices are defined by\[[39](https://arxiv.org/html/2610.12711#bib.bib4)\]

\(D𝒱\)i​i\\displaystyle\(D\_\{\\mathcal\{V\}\}\)\_\{ii\}=∑m=1KHi​m,\(Dℰ\)m​m=∑i=1KHi​m,\\displaystyle=\\sum\_\{m=1\}^\{K\}H\_\{im\},\\;\\;\(D\_\{\\mathcal\{E\}\}\)\_\{mm\}=\\sum\_\{i=1\}^\{K\}H\_\{im\},\(22\)with zero off\-diagonal entries\.mmindexes hyperedges\.

#### IV\-A3Hypergraph Message Passing

HMP is designed to assess each agent’s update using evidence from other agents connected through shared hyperedges\. Node\-to\-hyperedge aggregation summarizes their collective update patterns, while hyperedge\-to\-node propagation incorporates this context into each agent’s embedding\. The decoder then reconstructs pairwise similarities and hyperedge incidence from these embeddings\. This provide reconstruction errors that indicate departures from the modeled relational structure\.

Letν=0,1,…,L−1\\nu=0,1,\\ldots,L\-1represent HMP layers\. HMP alternates between node\-to\-hyperedge aggregation and hyperedge\-to\-node propagation following\[[40](https://arxiv.org/html/2610.12711#bib.bib3)\]

E\(ν\)\\displaystyle E^\{\(\\nu\)\}=ρ⁡\(Dℰ−1​H⊤​Z\(ν\)​Wℰ\(ν\)\),\\displaystyle=\\rho\(D\_\{\\mathcal\{E\}\}^\{\-1\}H^\{\\top\}Z^\{\(\\nu\)\}W\_\{\\mathcal\{E\}\}^\{\(\\nu\)\}\),\(23\)Z\(ν\+1\)\\displaystyle Z^\{\(\\nu\+1\)\}=ρ⁡\(D𝒱−1​H​E\(ν\)​W𝒱\(ν\)\),\\displaystyle=\\rho\(D\_\{\\mathcal\{V\}\}^\{\-1\}HE^\{\(\\nu\)\}W\_\{\\mathcal\{V\}\}^\{\(\\nu\)\}\),\(24\)whereZ\(0\)=XZ^\{\(0\)\}=XandE\(ν\),Z\(ν\+1\)∈ℝK×dE^\{\(\\nu\)\},Z^\{\(\\nu\+1\)\}\\in\\mathbb\{R\}^\{K\\times d\}\. The hidden activationρ\\rhocan be ReLU\. The output isZ=Z\(L\)∈ℝK×dZ=Z^\{\(L\)\}\\in\\mathbb\{R\}^\{K\\times d\}\.

#### IV\-A4Decoder Reconstruction

Let𝐀\\mathbf\{A\}denote graph adjacency\. Then, we have

𝐀i​j=\{1,if​i≠j​and​\(H​H⊤\)i​j\>0;0,otherwise\.\\displaystyle\\mathbf\{A\}\_\{ij\}=\\begin\{cases\}1,&\\text\{if \}i\\neq j\\text\{ and \}\(HH^\{\\top\}\)\_\{ij\}\>0;\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(25\)The decoder estimates their connectivity as

𝐀^i​j=sigmoid⁡\(zi⊤​zj\),i≠j\.\\displaystyle\\widehat\{\\mathbf\{A\}\}\_\{ij\}=\\operatorname\{sigmoid\}\(z\_\{i\}^\{\\top\}z\_\{j\}\),\\;\\;i\\neq j\.\(26\)Moreover, hyperedge membership can be reconstructed by

H^=sigmoid⁡\(Z​Wdec\+𝟏​bdec⊤\)∈\(0,1\)K×K,\\displaystyle\\widehat\{H\}=\\operatorname\{sigmoid\}\(ZW\_\{\\mathrm\{dec\}\}\+\\mathbf\{1\}b\_\{\\mathrm\{dec\}\}^\{\\top\}\)\\in\(0,1\)^\{K\\times K\},\(27\)whereWdec∈ℝd×KW\_\{\\mathrm\{dec\}\}\\in\\mathbb\{R\}^\{d\\times K\},bdec∈ℝKb\_\{\\mathrm\{dec\}\}\\in\\mathbb\{R\}^\{K\}, and𝟏∈ℝK\\mathbf\{1\}\\in\\mathbb\{R\}^\{K\}\.

DefineBCE⁡\(a,b\)=−a​log⁡b−\(1−a\)​log⁡\(1−b\)\\operatorname\{BCE\}\(a,b\)=\-a\\log b\-\(1\-a\)\\log\(1\-b\)\. The losses are given by

ℒH\\displaystyle\\mathcal\{L\}\_\{H\}=1K2​∑i=1K∑m=1KBCE⁡\(Hi​m,H^i​m\),\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{i=1\}^\{K\}\\sum\_\{m=1\}^\{K\}\\operatorname\{BCE\}\(H\_\{im\},\\widehat\{H\}\_\{im\}\),\(28\)ℒA\\displaystyle\\mathcal\{L\}\_\{A\}=1K⁡\(K−1\)​∑i≠jBCE⁡\(𝐀i​j,𝐀^i​j\)\.\\displaystyle=\\frac\{1\}\{K\(K\-1\)\}\\sum\_\{i\\neq j\}\\operatorname\{BCE\}\(\\mathbf\{A\}\_\{ij\},\\widehat\{\\mathbf\{A\}\}\_\{ij\}\)\.\(29\)A smoothness regularizer can be defined for neighboring representations, which gives

ℒsm=∑i≠j𝐀i​j​‖zi−zj‖22max⁡\(1,∑i≠j𝐀i​j\)\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{sm\}\}=\\frac\{\\sum\_\{i\\neq j\}\\mathbf\{A\}\_\{ij\}\\\|z\_\{i\}\-z\_\{j\}\\\|\_\{2\}^\{2\}\}\{\\max\(1,\\sum\_\{i\\neq j\}\\mathbf\{A\}\_\{ij\}\)\}\.\(30\)The training objective is given by

ℒGAE=λH​ℒH\+λA​ℒA\+λsm​ℒsm,\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{GAE\}\}=\\lambda\_\{H\}\\mathcal\{L\}\_\{H\}\+\\lambda\_\{A\}\\mathcal\{L\}\_\{A\}\+\\lambda\_\{\\mathrm\{sm\}\}\\mathcal\{L\}\_\{\\mathrm\{sm\}\},\(31\)

### IV\-BAlgorithm Design of Resilience Strategy

The novelty of HMP\-GAE lies in modeling Byzantine resilience in FedLLMs as a high\-order relational inference problem over LoRA updates\. Each agent’s model update is represented as a hypergraph node, allowing the server to assess its consistency with groups of agents instead of relying solely on isolated or pairwise comparisons\. HMP and reconstruction jointly capture these cross\-agent relationships, while historical consistency provides a temporal reference for assessing suspicious deviations\. The resilience estimates control federated aggregation, linking relational manipulation detection to the suppression of potentially corrupted model updates\.

Algorithm[1](https://arxiv.org/html/2610.12711#alg1)implements this strategy by combining decoder reconstruction errors with historical consistency to determine each agent’s aggregation weight\.

#### IV\-B1Historical Consistency and Training

The server can keep the checkpoint𝒞r=\(\{ukr,hkr−1\}k=1K,Hr\)\\mathcal\{C\}^\{r\}=\(\\\{u\_\{k\}^\{r\},h\_\{k\}^\{r\-1\}\\\}\_\{k=1\}^\{K\},H^\{r\}\), as well as the final embeddings and resilience scores\. For communication roundr\>0r\>0, it re\-encodes the preceding checkpoint with the current encoder parameters as

z¯kr=sg⁡\(\[EncΘ⁡\(𝒞r−1\)\]k\)\.\\displaystyle\\bar\{z\}\_\{k\}^\{r\}=\\operatorname\{sg\}\(\[\\operatorname\{Enc\}\_\{\\Theta\}\(\\mathcal\{C\}^\{r\-1\}\)\]\_\{k\}\)\.\(32\)Here,EncΘ⁡\(⋅\)\\operatorname\{Enc\}\_\{\\Theta\}\(\\cdot\)includes feature embedding and HMP,\[⋅\]k\[\\cdot\]\_\{k\}extracts a column\-vector node representation, andsg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\)stops gradients through the reference branch\.

Letϵ\>0\\epsilon\>0be a numerical stability constant\. The current and reference embeddings can be normalized as

z~kr\\displaystyle\\widetilde\{z\}\_\{k\}^\{r\}=zkr‖zkr‖2\+ϵ,z¯~kr=z¯kr‖z¯kr‖2\+ϵ\.\\displaystyle=\\frac\{z\_\{k\}^\{r\}\}\{\\\|z\_\{k\}^\{r\}\\\|\_\{2\}\+\\epsilon\},\\;\\;\\widetilde\{\\bar\{z\}\}\_\{k\}^\{r\}=\\frac\{\\bar\{z\}\_\{k\}^\{r\}\}\{\\\|\\bar\{z\}\_\{k\}^\{r\}\\\|\_\{2\}\+\\epsilon\}\.\(33\)Their historical discrepancy can be determined byχkr=‖z~kr−z¯~kr‖22\\chi\_\{k\}^\{r\}=\\\|\\widetilde\{z\}\_\{k\}^\{r\}\-\\widetilde\{\\bar\{z\}\}\_\{k\}^\{r\}\\\|\_\{2\}^\{2\}\. Based on the preceding resilience scores𝒵kr−1\\mathcal\{Z\}\_\{k\}^\{r\-1\}, the historical regularizer can be given by

ℒhistr=∑k=1K𝒵kr−1​χkrϵ\+∑k=1K𝒵kr−1\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{hist\}\}^\{r\}=\\frac\{\\sum\_\{k=1\}^\{K\}\\mathcal\{Z\}\_\{k\}^\{r\-1\}\\chi\_\{k\}^\{r\}\}\{\\epsilon\+\\sum\_\{k=1\}^\{K\}\\mathcal\{Z\}\_\{k\}^\{r\-1\}\}\.\(34\)Atr=0r=0, we setχk0=0\\chi\_\{k\}^\{0\}=0andℒhist0=0\\mathcal\{L\}\_\{\\mathrm\{hist\}\}^\{0\}=0\.

OverJJiterations, HMP\-GAE aims to minimize

𝒥r​\(Θ\)=ℒGAEr\+λhist​ℒhistr\+λreg​‖Θ‖22,\\displaystyle\\mathcal\{J\}^\{r\}\(\\Theta\)=\\mathcal\{L\}\_\{\\mathrm\{GAE\}\}^\{r\}\+\\lambda\_\{\\mathrm\{hist\}\}\\mathcal\{L\}\_\{\\mathrm\{hist\}\}^\{r\}\+\\lambda\_\{\\mathrm\{reg\}\}\\\|\\Theta\\\|\_\{2\}^\{2\},\(35\)whereλhist,λreg≥0\\lambda\_\{\\mathrm\{hist\}\},\\lambda\_\{\\mathrm\{reg\}\}\\geq 0\. The graphHrH^\{r\}and target𝐀r\\mathbf\{A\}^\{r\}remain fixed while features and embeddings are updated asΘ\\Theta\.

#### IV\-B2Resilience Scoring and Aggregation

The per\-agent structural errors present

eH,kr\\displaystyle e\_\{H,k\}^\{r\}=1K​∑m=1KBCE⁡\(Hk​mr,H^k​mr\),\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{m=1\}^\{K\}\\operatorname\{BCE\}\(H\_\{km\}^\{r\},\\widehat\{H\}\_\{km\}^\{r\}\),eA,kr\\displaystyle e\_\{A,k\}^\{r\}=1K−1​∑i≠kBCE⁡\(𝐀k​ir,𝐀^k​ir\)\.\\displaystyle=\\frac\{1\}\{K\-1\}\\sum\_\{i\\neq k\}\\operatorname\{BCE\}\(\\mathbf\{A\}\_\{ki\}^\{r\},\\widehat\{\\mathbf\{A\}\}\_\{ki\}^\{r\}\)\.\(36\)Non\-negative scores can be normalized by their round\-wise median as

𝒩r​\(\{ai\}i=1K\)k=akϵ\+mediani=1,…,K⁡ai\.\\displaystyle\\mathcal\{N\}\_\{r\}\(\\\{a\_\{i\}\\\}\_\{i=1\}^\{K\}\)\_\{k\}=\\frac\{a\_\{k\}\}\{\\epsilon\+\\operatorname\{median\}\_\{i=1,\\ldots,K\}a\_\{i\}\}\.\(37\)A scoreζkr\\zeta\_\{k\}^\{r\}can be obtained by

ζkr\\displaystyle\\zeta\_\{k\}^\{r\}=βH​𝒩r​\(\{eH,ir\}i=1K\)k\+βA​𝒩r​\(\{eA,ir\}i=1K\)k\\displaystyle=\\beta\_\{H\}\\mathcal\{N\}\_\{r\}\(\\\{e\_\{H,i\}^\{r\}\\\}\_\{i=1\}^\{K\}\)\_\{k\}\+\\beta\_\{A\}\\mathcal\{N\}\_\{r\}\(\\\{e\_\{A,i\}^\{r\}\\\}\_\{i=1\}^\{K\}\)\_\{k\}\+βhist​𝒩r​\(\{χir\}i=1K\)k,\\displaystyle\\quad\+\\beta\_\{\\mathrm\{hist\}\}\\mathcal\{N\}\_\{r\}\(\\\{\\chi\_\{i\}^\{r\}\\\}\_\{i=1\}^\{K\}\)\_\{k\},\(38\)whereβH,βA,βhist≥0\\beta\_\{H\},\\beta\_\{A\},\\beta\_\{\\mathrm\{hist\}\}\\geq 0are the non\-negative hyperparameters that control each component’s contribution\. Moreover, the resilience score is given as

𝒵kr=exp\(−ζkr/τ\),τ\>0\.\\displaystyle\\mathcal\{Z\}\_\{k\}^\{r\}=\\exp\(\-\\zeta\_\{k\}^\{r\}/\\tau\),\\qquad\\tau\>0\.\(39\)As shown in \([6](https://arxiv.org/html/2610.12711#S3.E6)\), larger inconsistencies reduce the agent’s contribution through

p~kr=pk​𝒵kr∑i=1Kpi​𝒵ir\.\\displaystyle\\tilde\{p\}\_\{k\}^\{r\}=\\frac\{p\_\{k\}\\mathcal\{Z\}\_\{k\}^\{r\}\}\{\\sum\_\{i=1\}^\{K\}p\_\{i\}\\mathcal\{Z\}\_\{i\}^\{r\}\}\.\(40\)The server then applies

ωr\+1=∑k=1Kp~kr​ω^​ζkr,\\displaystyle\\omega^\{r\+1\}=\\sum\_\{k=1\}^\{K\}\\tilde\{p\}\_\{k\}^\{r\}\\widehat\{\\omega\}\\zeta\_\{k\}^\{r\},\(41\)Reconstructed weight changes can be used for assessment, while aggregated factors are used for aggregation\. Thus, the server can storehkr=zkrh\_\{k\}^\{r\}=z\_\{k\}^\{r\},𝒵kr\\mathcal\{Z\}\_\{k\}^\{r\}, and𝒞r\\mathcal\{C\}^\{r\}, and broadcastωr\+1\\omega^\{r\+1\}\.

Algorithm 1HMP\-GAE for Byzantine\-Resilient FedLLMs0:

ωr\\omega^\{r\},

\{ω^​ζkr\}k=1K\\\{\\widehat\{\\omega\}\\zeta\_\{k\}^\{r\}\\\}\_\{k=1\}^\{K\},

\{pk\}k=1K\\\{p\_\{k\}\\\}\_\{k=1\}^\{K\},

\{hkr−1\}k=1K\\\{h\_\{k\}^\{r\-1\}\\\}\_\{k=1\}^\{K\}, and

Θ\\Theta\.

1:for

k=1,…,Kk=1,\\ldots,Kdo

2:Reconstruct

Uk,ℓrU\_\{k,\\ell\}^\{r\}and form

ukru\_\{k\}^\{r\}based on \([16](https://arxiv.org/html/2610.12711#S4.E16)\)\-\([17](https://arxiv.org/html/2610.12711#S4.E17)\)\.

3:\([18](https://arxiv.org/html/2610.12711#S4.E18)\)

→\\toinitial node features\.

4:endfor

5:\([21](https://arxiv.org/html/2610.12711#S4.E21)\), \([22](https://arxiv.org/html/2610.12711#S4.E22)\), and \([25](https://arxiv.org/html/2610.12711#S4.E25)\)

→\\toconstruct

HrH^\{r\}, the degree matrices, and

𝐀r\\mathbf\{A\}^\{r\}\.

6:for

t=1,…,Jt=1,\\ldots,Jdo

7:Obtain node features and apply the

LLHMP layers\.

8:Decode

𝐀^r\\widehat\{\\mathbf\{A\}\}^\{r\}and

H^r\\widehat\{H\}^\{r\}\.

9:if

r\>0r\>0then

10:Re\-encode

𝒞r−1\\mathcal\{C\}^\{r\-1\}and \([32](https://arxiv.org/html/2610.12711#S4.E32)\)–\([34](https://arxiv.org/html/2610.12711#S4.E34)\)

→\\toℒhistr\\mathcal\{L\}\_\{\\mathrm\{hist\}\}^\{r\}\.

11:else

12:Set

χkr=0\\chi\_\{k\}^\{r\}=0and

ℒhistr=0\\mathcal\{L\}\_\{\\mathrm\{hist\}\}^\{r\}=0\.

13:endif

14:\([35](https://arxiv.org/html/2610.12711#S4.E35)\)

→\\tominimize

𝒥r​\(Θ\)\\mathcal\{J\}^\{r\}\(\\Theta\)\.

15:endfor

16:Update final embeddings, reconstructions, and historical discrepancies with the updated

Θ\\Theta\.

17:\([36](https://arxiv.org/html/2610.12711#S4.E36)\)

→\\toper\-agent errors\.

18:Obtain

ζkr\\zeta\_\{k\}^\{r\},

𝒵kr\\mathcal\{Z\}\_\{k\}^\{r\}, and

p~kr\\tilde\{p\}\_\{k\}^\{r\}based on \([38](https://arxiv.org/html/2610.12711#S4.E38)\)–\([40](https://arxiv.org/html/2610.12711#S4.E40)\)\.

19:\([41](https://arxiv.org/html/2610.12711#S4.E41)\)

→\\toaggregate

ωr\+1\\omega^\{r\+1\}\.

20:Keep

𝒞r\\mathcal\{C\}^\{r\},

hkr=zkrh\_\{k\}^\{r\}=z\_\{k\}^\{r\}, and

𝒵kr\\mathcal\{Z\}\_\{k\}^\{r\}\.

21:Broadcast

ωr\+1\\omega^\{r\+1\}\.

### IV\-CComplexity Analysis

LetΦ⁡\(H\)\\Phi\(H\)denote the number of non\-zero incidence entries, andΦ⁡\(H\)=K⁡\(s\+1\)\\Phi\(H\)=K\(s\+1\)\. Each HMP layer costs𝒪⁡\(K​d2\+Φ⁡\(H\)​d\)\\mathcal\{O\}\(Kd^\{2\}\+\\Phi\(H\)d\), while dense incidence and adjacency decoding each cost𝒪⁡\(K2​d\)\\mathcal\{O\}\(K^\{2\}d\)\. Given an input projection fromD\+dD\+dtodd, historical re\-encoding and final scoring pass duringJJsteps cost

𝒪​\(CLOSE​\(J\+1\)​\[K​D​d\+L⁡\(K​d2\+K⁡\(s\+1\)​d\)\+K2​d\]​\)\.\\displaystyle\\mathcal\{O\}\\hbox\{$\\left\(\\vbox to0\.0pt\{\}\\right\.$\}\(J\\\!\+\\\!1\)\[KDd\\\!\+\\\!L\(Kd^\{2\}\\\!\\\!\+\\\!K\(s\\\!\+\\\!1\)d\)\\\!\+\\\!K^\{2\}d\]\\hbox\{$\\left\)\\vbox to0\.0pt\{\}\\right\.$\}\.\(42\)The additionalK​d2Kd^\{2\}feature\-processing cost is absorbed by the encoder term forL≥1L\\geq 1\.

Explicit LoRA product reconstruction costs𝒪⁡\(K​∑ℓ∈𝒮aℓ​bℓ​qℓ\)\\mathcal\{O\}\(K\\sum\_\{\\ell\\in\\mathcal\{S\}\}a\_\{\\ell\}b\_\{\\ell\}q\_\{\\ell\}\)\. The neighbor search with linear\-time selection and adjacency construction can cost𝒪⁡\(K2​d\+K​\(s\+1\)2\)\\mathcal\{O\}\(K^\{2\}d\+K\(s\+1\)^\{2\}\), and aggregation costs𝒪⁡\(K​P\)\\mathcal\{O\}\(KP\), wherePPcontains the total number of scalar entries ofω\\omegain \([3](https://arxiv.org/html/2610.12711#S3.E3)\)\. The total per\-round server cost is

𝒪​\(CLOSE\\displaystyle\\mathcal\{O\}\\hbox\{$\\left\(\\vbox to0\.0pt\{\}\\right\.$\}K​∑ℓ∈𝒮aℓ​bℓ​qℓ\+K​P\+K2​d\+K​\(s\+1\)2\+\(J\+1\)\\displaystyle K\\sum\_\{\\ell\\in\\mathcal\{S\}\}a\_\{\\ell\}b\_\{\\ell\}q\_\{\\ell\}\+KP\+K^\{2\}d\+K\(s\+1\)^\{2\}\+\(J\+1\)×\[K​D​d\+L⁡\(K​d2\+K⁡\(s\+1\)​d\)\+K2​d\]​\)\.\\displaystyle\\times\[KDd\+L\(Kd^\{2\}\+K\(s\+1\)d\)\+K^\{2\}d\]\\hbox\{$\\left\)\\vbox to0\.0pt\{\}\\right\.$\}\.\(43\)Givenssanddd, the sparse HMP propagation scales linearly withKK\.

## VPerformance Evaluation

This section presents the implementation of HMP\-GAE in FedLLMs under Byzantine attacks\. The source code for HMP\-GAE is available at:[https://github\.com/GuangLun2000/HMP\-GAE](https://github.com/GuangLun2000/HMP-GAE)\.

### V\-AExperimental Implementation

The experimental evaluation is conducted on a Linux workstation equipped with an NVIDIA RTX PRO 6000 Blackwell GPU with 96 GB of memory\. The framework is implemented in Python 3\.12 with PyTorch 2\.10\. Two representative text classification datasets\[[41](https://arxiv.org/html/2610.12711#bib.bib40)\]are adopted to evaluate the proposed HMP\-GAE framework:

- •The first is theAG News dataset, which consists of four topical categories, namely World, Sports, Business, and Sci/Tech, containing 120,000 training samples and 7,600 test samples\.
- •The second one is theYahoo\! Answers dataset, a large\-scale corpus for topic classification that includes 10 categories, with approximately 1\.4 million training instances and 60,000 test samples\.

These datasets differ in label granularity and corpus scale, helping assess the Byzantine\-resilient performance under distinct semantic distributions and learning conditions\.

Two pretrained backbones are deployed at the agents to help examine the effect of model capacity on Byzantine resilience\.

- •Qwen\-2\.5\[[42](https://arxiv.org/html/2610.12711#bib.bib41)\]is a model with approximately 0\.5 billion parameters, pretrained on 18 trillion tokens with broad multilingual coverage and a context length of up to 32K tokens\.
- •Llama\-3\.2\[[43](https://arxiv.org/html/2610.12711#bib.bib30)\]is a model with around 1\.2 billion parameters, pretrained on up to 9 trillion tokens from multilingual online data and enhanced through knowledge distillation from larger Llama models, with a context length of up to 128K tokens\.

To enable efficient adaptation of these backbones in the federated setting, we employ LoRA, where trainable low\-rank matrices are injected into the attention and feedforward projections while the pretrained weights remain frozen\. Consequently, each agent optimizes and uploads only the flattened low\-rank update, which constitutes both the communication payload and the input to the proposed resilience mechanism at the server\.

TABLE I:Global model performance of the proposed HMP\-GAE, compared to Multi\-Krum, FoolsGold, and FLTrust methods\. Accuracy and CSEmeanare averaged over 50 rounds, while CSEfinaland PPL are evaluated based on the final global model\.
### V\-BPerformance Metrics

HMP\-GAE is evaluated using three complementary performance metrics, namely prediction accuracy, classification semantic entropy \(CSE\), and perplexity \(PPL\), to comprehensively assess its semantic resilience, language modeling capability, and task\-specific prediction in FedLLMs\.

- •Prediction accuracy is adopted to evaluate how precisely a FedLLM can predict the target outputs, i\.e\., class labels, on unseen data after federated aggregation\. It is computed as the ratio of correctly estimated samples to the total number of evaluated samples\. A higher estimation accuracy indicates that the aggregated FedLLM can correctly preserve task\-specific knowledge learned from distributed agents and produce more reliable predictions under data heterogeneity and potentially malicious updates\.
- •CSE is employed to quantify the semantic uncertainty of FedLLMs, which evaluates whether the model assigns predictive mass consistently to a single meaning\. It is the mean Shannon entropy of the predictive class distribution over the evaluation set, where the class labels serve as semantic equivalence classes and provide a generation\-free surrogate of semantic entropy\. Lower CSE indicates that the FedLLM produces semantically stable and confident predictions; higher CSE suggests greater semantic dispersion, uncertainty, or potential Byzantine attacks\.
- •PPL measures how well the aggregated global model predicts a sequence of tokens\. A lower perplexity indicates that the FedLLM assigns higher probability to the correct token sequence, suggesting stronger language understanding, generalization, and more effective aggregation across distributed agents\.

### V\-CResilience Performance

The proposed HMP\-GAE is compared with three representative robust\-aggregation defenses under an identical federated configuration, where the aggregation rule at the server is the only variable\.

- •Multi\-Krum\[[44](https://arxiv.org/html/2610.12711#bib.bib42)\]scores each update by the sum of squared Euclidean distances to its nearest neighbors, and averages the several updates with the lowest scores, thereby retaining only the contributions that reside in the densest region of the parameter space\.
- •FoolsGold\[[45](https://arxiv.org/html/2610.12711#bib.bib44)\]identifies colluding agents by the pairwise cosine similarity of their historical updates, and attenuates the learning rate of the agents whose update directions are persistently aligned with each other\.
- •FLTrust\[[46](https://arxiv.org/html/2610.12711#bib.bib43)\]maintains a clean root dataset at the server to compute a reference update, and weighs each agent by the cosine similarity between its update and this reference after normalizing the update magnitudes\.

#### V\-C1Performance on The Two Datasets

Table[I](https://arxiv.org/html/2610.12711#S5.T1)compares HMP\-GAE with Multi\-Krum, FoolsGold, and FLTrust onAG NewsandYahoo\! Answers\. HMP\-GAE achieves the highest detection accuracy across both datasets and model backbones\. OnAG News, HMP\-GAE achieves an accuracy of 88\.95% with Qwen\-2\.5, outperforming Multi\-Krum, FoolsGold, and FLTrust by 3\.27%, 0\.95%, and 1\.48%, respectively\. With Llama\-3\.2, HMP\-GAE achieves 91\.44%, improving over the same baselines by 2\.28%, 1\.81%, and 1\.21%\. OnYahoo\! Answers, HMP\-GAE achieves 61\.12% with Qwen\-2\.5 and 66\.54% with Llama\-3\.2, corresponding to gains of 6\.99%, 0\.72%, and 4\.58%, and 4\.02%, 1\.03%, and 1\.90%, respectively\. These gains confirm that our hypergraph message\-passing structure sufficiently captures higher\-order relationships among multiple connected entities, providing richer contextual evidence for distinguishing corrupted from non\-corrupted outputs\. The graph autoencoder’s reconstruction objective can further encourage representations that preserve shared relational structure, helping separate consistent patterns from anomalous ones and thereby supporting more accurate detection\.

HMP\-GAE achieves the lowest CSEmeanand CSEfinalin every evaluated setting\. OnAG News, its CSEmean/CSEfinalvalues are 0\.1267/0\.0395 with Qwen\-2\.5 and 0\.1067/0\.0258 with Llama\-3\.2, representing reductions of 21\.21%/0\.50% and 30\.03%/11\.03% relative to the strongest baseline for each metric\. OnYahoo\! Answers, the corresponding values are 0\.6378/0\.2380 and 0\.5095/0\.2291, yielding reductions of 8\.60%/21\.71% and 30\.75%/44\.08%, respectively\. Fig\.[3](https://arxiv.org/html/2610.12711#S5.F3)further shows the global model CSE over 50 communication rounds, where a lower CSEmeanindicates reduced semantic uncertainty throughout training and a lower CSEfinalshows that this advantage persists in the final global model\. Together with the highest accuracy, this indicates that HMP\-GAE produces more decisive classifications aligned with the correct labels\. Sharper discrimination among the semantic classes is therefore a plausible explanation for this advantage, rather than lower entropy alone being taken as evidence of correctness\.

Hypergraph message passing can contribute to this reduction by combining evidence across jointly connected entities, allowing group\-level context to resolve ambiguities that individual representations may retain\. The autoencoder’s reconstruction objective complements this process by building structurally consistent latent representations, providing a plausible basis for the lower semantic uncertainty observed alongside improved accuracy\.

In addition, HMP\-GAE achieves the lowest final\-model PPL across all four settings\. OnAG News, its PPL is 69\.49 with Qwen\-2\.5 and 46\.52 with Llama\-3\.2, representing reductions of 16\.43%, 11\.83%, and 8\.66%, and 32\.74%, 22\.52%, and 13\.34% relative to Multi\-Krum, FoolsGold, and FLTrust, respectively\. OnYahoo\! Answers, HMP\-GAE reduces PPL to 1061\.74 with Qwen\-2\.5 and 525\.65 with Llama\-3\.2, improving over the same baselines by 15\.57%, 31\.47%, and 5\.88%, and 23\.78%, 29\.05%, and 4\.98%, respectively\. As lower PPL indicates that the final global model assigns higher likelihood to the evaluation text, the performance enhancement of PPL is consistent with HMP\-GAE retaining useful LLMs information during federated training\. Moreover, the graph autoencoder’s reconstruction reinforces the retention of these shared patterns in the learned representations, which supports the final global model’s language\-modeling performance and lower PPL\.

![Refer to caption](https://arxiv.org/html/2610.12711v1/cse_global_four_figuers.png)Fig\. 3:Global model CSE of four defenses under the Byzantine attack across 50 communication rounds\.![Refer to caption](https://arxiv.org/html/2610.12711v1/cse_local_four_figuers.png)Fig\. 4:Local CSE averaged over all benign agents under four defenses against the Byzantine attack across 50 communication rounds\.TABLE II:Local accuracy \(Acc\.\) and local CSE averaged over all benign local agents under the Byzantine attack\.![Refer to caption](https://arxiv.org/html/2610.12711v1/trust_weight_eight.png)Fig\. 5:Per\-user trust weight of four different defenses over 50 communication rounds under the Byzantine attack, with the Qwen\-2\.5 and Llama\-3\.2 backbones onAG News dataset\.Table[II](https://arxiv.org/html/2610.12711#S5.T2)presents that HMP\-GAE achieves the highest average local accuracy and the lowest average local CSE among benign agents under the Byzantine attack, where Fig\.[4](https://arxiv.org/html/2610.12711#S5.F4)shows the variation of local models’ CSE over 50 rounds\. OnAG News, the local accuracies of HMP\-GAE are 85\.00% and 87\.69% with Qwen\-2\.5 and Llama\-3\.2, respectively, surpassing the strongest accuracy baseline in each setting by 0\.24% and 0\.59%\. The corresponding local CSE values decrease to 0\.1257 and 0\.1085, representing reductions of 15\.92% and 13\.61% relative to the lowest\-CSE baseline\. OnYahoo\! Answers, HMP\-GAE has local accuracies of 50\.52% and 55\.03%, exceeding the strongest baselines by 0\.89% and 0\.77%, while reducing local CSE to 0\.5531 and 0\.4613, corresponding to reductions of 6\.73% and 21\.95%, respectively\. This indicates that HMP\-GAE’s benefits extend to benign local agents, improving their average prediction accuracy while reducing semantic uncertainty under attack\.

### V\-DPer\-user Trust Weight

Fig\.[5](https://arxiv.org/html/2610.12711#S5.F5)presents the trust weights assigned to five benign users \(C0 to C4\) and two malicious users \(C5 and C6\) onAG News\. With Qwen\-2\.5 and Llama\-3\.2, HMP\-GAE reduces the two malicious weights to approximately zero and maintains this suppression throughout the remaining rounds, while assigning stable, positive weights to all five benign users\. This clear separation demonstrates the persistent identification of malicious users without excluding benign participants\.

Multi\-Krum, however, intermittently assigns malicious users weights of approximately 0\.20, equal to those of selected benign users, indicating that malicious updates remain eligible for aggregation in some rounds\. FLTrust also fails to separate the two groups: malicious users retain substantial, fluctuating weights that overlap with those of benign ones, particularly with Llama\-3\.2\. Although FoolsGold suppresses two malicious users, it also reduces the benign user C0’s weight to approximately zero for most rounds under both model backbones, revealing a persistent false alarm that excludes a legitimate participant’s contribution\. This confirms HMP\-GAE’s ability to jointly suppress malicious contributions and retain benign participation, where hypergraph message passing successfully captures higher\-order relationships among users, providing collective evidence for distinguishing malicious behavior from benign heterogeneity\. The graph autoencoder’s reconstruction objective further encourages representations that preserve consistent relational patterns, supporting the sustained separation observed in the trust weights\.

Fig\. 6:Per\-class PPL of the final global model under four different defenses against the Byzantine attack, with the Qwen\-2\.5 and Llama\-3\.2 backbones onAG News datasetandYahoo\! Answers dataset\.TABLE III:Ablation study of HMP\-GAE under the Byzantine attack, comparing the full HMP\-GAE framework with a GAE\-based defense without HMP and with HMP\-GAE without hyperedge reconstruction\. Accuracy and CSE are averaged over 50 rounds, and PPL is evaluated on the final global model\.
### V\-EPer\-class PPL

In Fig\.[6](https://arxiv.org/html/2610.12711#S5.F6), the final global model is evaluated across four classes inAG Newsand ten classes inYahoo\! Answersusing Qwen\-2\.5 and Llama\-3\.2\. Specifically, HMP\-GAE achieves the lowest PPL in all fourAG Newsclasses with both model backbones, demonstrating improvements across the full range of topics\. OnYahoo\! Answers, HMP\-GAE obtains the lowest PPL in most classes and the lowest mean per\-class PPL with both backbones, although several class\-specific exceptions remain\. With Qwen\-2\.5, FoolsGold achieves lower PPL on Science & Mathematics, while FLTrust performs slightly better on Health and Sports and more clearly better on Politics & Government\.

With Llama\-3\.2, FLTrust achieves lower PPL on Sports and Politics & Government while Multi\-Krum also slightly outperforms HMP\-GAE on Sports\. These exceptions may reflect differences in class representation across users and in the topic\-specific capabilities of the two model backbones\. In particular, assigning trust at the user level can reduce the contribution of an untrustworthy user who nevertheless supplies some useful information for a particular class, creating a trade\-off between suppressing malicious contributions and retaining class\-specific knowledge\. Hypergraph message passing and autoencoder reconstruction can favor patterns supported by consistent relationships among users, but such patterns may represent less widely shared topics less effectively\. Therefore, the lowest PPL in every individual class is not guaranteed, even when the method improves overall performance\. HMP\-GAE’s lowest mean per\-class PPL across all four settings supports its aggregate benefit despite these localized trade\-offs\.

### V\-FAblation Study

Table[III](https://arxiv.org/html/2610.12711#S5.T3)evaluates the contributions of HMP and hyperedge reconstruction\. Specifically, the proposed HMP\-GAE achieves the highest accuracy and lowest CSE and PPL across Qwen\-2\.5 and Llama\-3\.2 models, as well as the two datasets\. Compared with the GAE\-based defense, HMP\-GAE improves accuracy by0\.26%0\.26\\%\-1\.39%1\.39\\%, while reducing CSE by27\.43%27\.43\\%\-54\.48%54\.48\\%, and lowering PPL by8\.82%8\.82\\%\-35\.49%35\.49\\%\. In particular, for Qwen\-2\.5 onYahoo\! Answers, accuracy increases from59\.73%59\.73\\%to61\.12%61\.12\\%\. For Llama\-3\.2, the largest CSE reduction occurs onAG News, while the largest PPL reduction occurs onYahoo\! Answers\. This performance enhancement further confirms the importance of HMP for capturing relationships among multiple \(benign and malicious\) agents beyond pairwise graph interactions\. Such relational information helps assess Byzantine\-inducing model updates that may appear consistent individually but deviate from broader group patterns\.

Furthermore, compared with HMP\-GAE without Hyperedge Reconstruction, the proposed HMP\-GAE defense improves accuracy by0\.21%0\.21\\%\-1\.11%1\.11\\%, reduces CSE by5\.00%5\.00\\%\-14\.38%14\.38\\%, and lowers PPL by4\.48%4\.48\\%\-15\.25%15\.25\\%\. The most significant accuracy gain and CSE reduction are observed for Qwen\-2\.5 onYahoo\! Answers\. For Llama\-3\.2 on the same dataset, hyperedge reconstruction reduces PPL from620\.23620\.23to525\.65525\.65\. This is because hyperedge reconstruction in HMP\-GAE can flag model updates associated with unusually high classification uncertainty before relational assessment\. Reducing the influence further limits the propagation of unreliable information during HMP\.

## VIConclusions

In this paper, we proposed HMP\-GAE, a resilience framework that mitigates Byzantine\-inducing updates in FedLLMs through self\-healing federated aggregation\. We formulated a defensive optimization problem for benign users to maximize global task accuracy while filtering out malicious, semantically manipulated contributions during aggregation\. HMP\-GAE represents each agent’s LoRA\-fine\-tuned model update as a hypergraph node, with hyperedges capturing higher\-order semantic and contextual relationships among multiple updates beyond pairwise correlations\. The server iteratively passes messages across these hyperedges to refine benign\-update embeddings, enabling collective reasoning to identify and suppress Byzantine\-inducing or semantically inconsistent contributions\. Experiments on AG News and Yahoo\! Answers using Qwen\-2\.5 and Llama\-3\.2 demonstrated that HMP\-GAE consistently achieved higher accuracy and lower classification semantic entropy and overall perplexity than the evaluated baselines\. Trust\-weight analysis further showed that HMP\-GAE persistently suppressed malicious users’ contributions while preserving benign participation, with benefits extending to benign agents’ local performance\.

## References

- \[1\]H\. Cai, H\. Dong, H\. Wang, K\. Li, and O\. B\. Akan\(2025\)Graph representation\-based model poisoning on federated llms in cyberedge networks\.arXiv preprint arXiv:2507\.01694\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p1.1)\.
- \[2\]H\. Li, H\. Madhukumar, N\. Methley, X\. Chen, Y\. Wu, J\. Parra\-Ullauri, V\. Sharma, J\. Lee, A\. R\. Koblitz, M\. Andrews,et al\.\(2025\)Future factories with 6g: agentic ai and cyber\-physical digital twins\.IEEE Internet of Things Journal\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p1.1)\.
- \[3\]Y\. Qiao, H\. Le, A\. Raha, P\. Tran, A\. Adhikary, M\. Zhang, L\. X\. Nguyen, E\. Huh, D\. Niyato, and C\. S\. Hong\(2025\)A survey on foundation models for personalized federated intelligence\.ACM Computing Surveys\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p1.1)\.
- \[4\]Y\. Cheng, W\. Zhang, Z\. Zhang, C\. Zhang, S\. Wang, and S\. Mao\(2024\)Towards federated large language models: motivations, methods, and future directions\.IEEE Communications Surveys & Tutorials\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p1.1)\.
- \[5\]K\. Li, Z\. Zhang, A\. Pourkabirian, W\. Ni, F\. Dressler, and O\. B\. Akan\(2025\)Towards resilient federated learning in cyberedge networks: recent advances and future trends\.arXiv preprint arXiv:2504\.01240\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p1.1)\.
- \[6\]Y\. Yao, J\. Zhang, J\. Wu, C\. Huang, Y\. Xia, T\. Yu, R\. Zhang, S\. Kim, R\. Rossi, A\. Li,et al\.\(2024\)Federated large language models: current progress and future directions\.arXiv preprint arXiv:2409\.15723\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p1.1)\.
- \[7\]T\. Webb, S\. S\. Mondal, and I\. Momennejad\(2025\)A brain\-inspired agentic architecture to improve planning with llms\.Nature Communications16\(1\),pp\. 8633\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p1.1)\.
- \[8\]H\. Yang, J\. Chen, M\. Siew, T\. Lorido\-Botran, and C\. Joe\-Wong\(2025\)Llm\-powered decentralized generative agents with adaptive hierarchical knowledge graph for cooperative planning\.arXiv preprint arXiv:2502\.05453\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p1.1)\.
- \[9\]H\. Cai, K\. Li, H\. Wang, H\. Dong, Y\. Li, F\. Dressler, and O\. B\. Akan\(2026\)Graph representation learning augmented model manipulation on federated fine\-tuning of llms\.arXiv preprint arXiv:2605\.07961\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p1.1)\.
- \[10\]H\. Cai\(2024\)Securing billion bluetooth devices leveraging learning\-based techniques\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 23731–23732\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p2.1)\.
- \[11\]Z\. Wang, Y\. Zhou, Y\. Shi, and K\. B\. Letaief\(2025\)Federated fine\-tuning for pre\-trained foundation models over wireless networks\.IEEE Transactions on Wireless Communications\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p2.1)\.
- \[12\]J\. Hu, D\. Wang, Z\. Wang, X\. Pang, H\. Xu, J\. Ren, and K\. Ren\(2024\)Federated large language model: solutions, challenges and future directions\.IEEE Wireless Communications\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p2.1)\.
- \[13\]H\. Cai, H\. Dong, H\. Wang, K\. Li, S\. Zou, and O\. B\. Akan\(2026\)Graph representation\-based model poisoning on the heterogeneous internet of agents\.In2026 International Wireless Communications and Mobile Computing \(IWCMC\),pp\. 956–961\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p2.1)\.
- \[14\]C\. Ren, H\. Yu, H\. Peng, X\. Tang, B\. Zhao, L\. Yi, A\. Z\. Tan, Y\. Gao, A\. Li, X\. Li,et al\.\(2025\)Advances and open challenges in federated foundation models\.IEEE Communications Surveys & Tutorials28,pp\. 2087–2126\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p3.1)\.
- \[15\]M\. P\. Uddin, Y\. Xiang, M\. Hasan, J\. Bai, Y\. Zhao, and L\. Gao\(2025\)A systematic literature review of robust federated learning: issues, solutions, and future research directions\.ACM Computing Surveys57\(10\),pp\. 1–62\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p3.1)\.
- \[16\]H\. Cai, J\. Zhan, Z\. Lin, and X\. Cai\(2026\)Poster: graph learning\-based update manipulation attack on federated fine\-tuning of llms over wireless networks\.InProceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services Companion,pp\. 27–28\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p3.1)\.
- \[17\]W\. Huang, M\. Ye, Z\. Shi, G\. Wan, H\. Li, B\. Du, and Q\. Yang\(2024\)Federated learning for generalization, robustness, fairness: a survey and benchmark\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(12\),pp\. 9387–9406\.Cited by:[§I](https://arxiv.org/html/2610.12711#S1.p4.1)\.
- \[18\]Y\. Xia, Z\. Tan, P\. Zhang, L\. Yu, S\. Dong, and Y\. Liu\(2026\)Communication\-efficient and byzantine\-tolerant federated learning for multi\-access edge computing\.IEEE Transactions on Mobile Computing\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p1.1)\.
- \[19\]J\. Wang, R\. Wang, Z\. Li, F\. Zhang, J\. Li, and X\. Li\(2026\)A byzantine\-robust secure federated learning scheme in heterogeneous data\.IEEE Transactions on Information Forensics and Security\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p1.1)\.
- \[20\]Y\. Li, S\. Bhujel, C\. Lira, N\. Wang, and Y\. Xiao\(2026\)Model\-consistent byzantine\-resilient decentralized federated learning for collaborative missions\.arXiv preprint arXiv:2608\.22552\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p2.1)\.
- \[21\]M\. Shen, B\. Peng, Y\. Zhao, Y\. Qin, M\. Li, Q\. Li, and L\. Zhu\(2026\)Byzantine\-robust asynchronous federated learning via feature fingerprinting\.IEEE Transactions on Information Forensics and Security\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p2.1)\.
- \[22\]Y\. Xia, C\. Hofmeister, M\. Egger, and R\. Bitar\(2026\)Perfect privacy for discriminator\-based byzantine\-resilient federated learning\.IEEE Transactions on Information Forensics and Security\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p3.1)\.
- \[23\]H\. Zhang, X\. He, J\. Wu, and Q\. Wang\(2026\)Building trust beyond update divergence: dual\-refined aggregation for byzantine\-robust federated learning\.IEEE Transactions on Information Forensics and Security\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p3.1)\.
- \[24\]G\. Zhu, W\. Shen, Z\. Liu, J\. Qin, and J\. Ma\(2026\)BPFLH: byzantine\-robust privacy\-preserving federated learning for heterogeneous data\.IEEE Transactions on Dependable and Secure Computing\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p4.1)\.
- \[25\]X\. Qin, X\. Yang, and X\. Tang\(2026\)Efficient byzantine\-robust privacy\-preserving federated learning via dimension compression\.IEEE Transactions on Information Forensics and Security\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p4.1)\.
- \[26\]X\. Tang, M\. Li, M\. Shen, J\. Kang, L\. Zhu, Z\. Liu, G\. Yang, D\. Niyato, and R\. H\. Deng\(2025\)Roby: a byzantine\-robust and privacy\-preserving serverless federated learning framework\.IEEE Transactions on Information Forensics and Security\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p5.1)\.
- \[27\]A\. Song, T\. Zhang, K\. Cheng, Y\. Cao, X\. Zhu, and Y\. Shen\(2025\)Byzantine\-robust federated learning framework via a server\-client defense mechanisms\.IEEE Internet of Things Journal12\(14\),pp\. 29073–29088\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p5.1)\.
- \[28\]G\. Wang, H\. Wu, Q\. Li, Z\. Liu, J\. Zhao, K\. Lam, Q\. Dong, Y\. Yang, and L\. Zhou\(2025\)PDFL: a privacy\-enhancing and robust poisoning defense federated learning scheme\.IEEE Transactions on Information Forensics and Security21,pp\. 608–623\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p6.1)\.
- \[29\]K\. Xiao, Q\. Wang, and C\. Anagnostopoulos\(2025\)FLgym: toward robust and byzantine\-resilient federated learning\.IEEE Transactions on Information Forensics and Security21,pp\. 404–416\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p6.1)\.
- \[30\]B\. Zhang, M\. Fang, Z\. Liu, B\. Yi, P\. Zhou, Y\. Wang, T\. Li, and Z\. Liu\(2025\)Practical framework for privacy\-preserving and byzantine\-robust federated learning\.IEEE Transactions on Information Forensics and Security21,pp\. 61–75\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p7.1)\.
- \[31\]J\. Yu, H\. Zhang, Q\. Xia, Y\. Zou, and Y\. Liu\(2025\)LPP\-fl: a lightweight privacy\-preserving federated learning against byzantine attacks on non\-iid data\.IEEE Transactions on Information Forensics and Security\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p7.1)\.
- \[32\]C\. Herath, Y\. Rahulamathavan, V\. De Silva, and S\. Lambotharan\(2025\)DSFL: a dual\-server byzantine\-resilient federated learning framework via group\-based secure aggregation\.arXiv preprint arXiv:2509\.08449\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p8.1)\.
- \[33\]Y\. Tao, Z\. Zhang, D\. Wang, D\. Yu, X\. Cheng, and F\. Dressler\(2026\)Byzantine\-resilient federated learning under heterogeneity and heavy tails\.IEEE Transactions on Networking\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p8.1)\.
- \[34\]A\. Yazdinejad, A\. Dehghantanha, H\. Karimipour, G\. Srivastava, and R\. M\. Parizi\(2024\)A robust privacy\-preserving federated learning model against model poisoning attacks\.IEEE Transactions on Information Forensics and Security19,pp\. 6693–6708\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p9.1)\.
- \[35\]C\. Dong, J\. Weng, M\. Li, J\. Liu, Z\. Liu, Y\. Cheng, and S\. Yu\(2023\)Privacy\-preserving and byzantine\-robust federated learning\.IEEE Transactions on Dependable and Secure Computing21\(2\),pp\. 889–904\.Cited by:[§II\-A](https://arxiv.org/html/2610.12711#S2.SS1.p9.1)\.
- \[36\]X\. He, Y\. Tian, Y\. Sun, N\. V\. Chawla, T\. Laurent, Y\. LeCun, X\. Bresson, and B\. Hooi\(2024\)G\-retriever: retrieval\-augmented generation for textual graph understanding and question answering\.Advances in Neural Information Processing Systems37,pp\. 132876–132907\.Cited by:[§III\-A](https://arxiv.org/html/2610.12711#S3.SS1.p2.1)\.
- \[37\]G\. Yan, L\. Xie, Q\. Shen, Y\. Fang, and Z\. Wu\(2026\)FedSRD: sparsify\-reconstruct\-decompose for communication\-efficient federated large language models fine\-tuning\.InProceedings of the ACM Web Conference 2026,pp\. 5087–5098\.Cited by:[§III\-B](https://arxiv.org/html/2610.12711#S3.SS2.p2.1)\.
- \[38\]Z\. Wang, Z\. Shen, Y\. He, G\. Sun, H\. Wang, L\. Lyu, and A\. Li\(2024\)Flora: federated fine\-tuning large language models with heterogeneous low\-rank adaptations\.Advances in Neural Information Processing Systems37,pp\. 22513–22533\.Cited by:[§III\-B](https://arxiv.org/html/2610.12711#S3.SS2.p2.1)\.
- \[39\]X\. Zhang, J\. Ma, J\. Guo, W\. Hu, Z\. Qi, F\. Hui, J\. Yang, and Y\. Zhang\(2025\)Hypergct: a dynamic hyper\-gnn\-learned geometric constraint for 3d registration\.In2025 IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 24750–24759\.Cited by:[§IV\-A2](https://arxiv.org/html/2610.12711#S4.SS1.SSS2.p4.1)\.
- \[40\]L\. Telyatnikov, M\. S\. Bucarelli, G\. Bernardez, O\. Zaghen, S\. Scardapane, and P\. Lió\(2025\)Hypergraph neural networks through the lens of message passing: a common perspective to homophily and architecture design\.Transactions on Machine Learning Research\.Cited by:[§IV\-A3](https://arxiv.org/html/2610.12711#S4.SS1.SSS3.p2.1)\.
- \[41\]X\. Zhang, J\. Zhao, and Y\. LeCun\(2015\)Character\-level convolutional networks for text classification\.Advances in neural information processing systems28\.Cited by:[§V\-A](https://arxiv.org/html/2610.12711#S5.SS1.p1.1)\.
- \[42\]B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu,et al\.\(2024\)Qwen2\. 5\-coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[1st item](https://arxiv.org/html/2610.12711#S5.I2.i1.p1.1),[TABLE I](https://arxiv.org/html/2610.12711#S5.T1.11.1.4.1.1),[TABLE II](https://arxiv.org/html/2610.12711#S5.T2.6.1.3)\.
- \[43\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[2nd item](https://arxiv.org/html/2610.12711#S5.I2.i2.p1.1),[TABLE I](https://arxiv.org/html/2610.12711#S5.T1.11.1.5.1.1),[TABLE II](https://arxiv.org/html/2610.12711#S5.T2.6.1.4)\.
- \[44\]P\. Blanchard, E\. M\. E\. Mhamdi, R\. Guerraoui, and J\. Stainer\(2017\)Byzantine\-Tolerant Machine Learning\.arXiv preprint arXiv:1703\.02757\(arXiv:1703\.02757\)\.External Links:1703\.02757Cited by:[1st item](https://arxiv.org/html/2610.12711#S5.I4.i1.p1.1),[TABLE I](https://arxiv.org/html/2610.12711#S5.T1.11.3.2),[TABLE I](https://arxiv.org/html/2610.12711#S5.T1.11.7.2),[TABLE II](https://arxiv.org/html/2610.12711#S5.T2.6.3.2),[TABLE II](https://arxiv.org/html/2610.12711#S5.T2.6.7.2)\.
- \[45\]C\. Fung, C\. J\. M\. Yoon, and I\. Beschastnikh\(2020\)Mitigating Sybils in Federated Learning Poisoning\.arXiv preprint arXiv:1808\.04866\(arXiv:1808\.04866\)\.External Links:1808\.04866Cited by:[2nd item](https://arxiv.org/html/2610.12711#S5.I4.i2.p1.1),[TABLE I](https://arxiv.org/html/2610.12711#S5.T1.11.4.1),[TABLE I](https://arxiv.org/html/2610.12711#S5.T1.11.8.1),[TABLE II](https://arxiv.org/html/2610.12711#S5.T2.6.4.1),[TABLE II](https://arxiv.org/html/2610.12711#S5.T2.6.8.1)\.
- \[46\]X\. Cao, M\. Fang, J\. Liu, and N\. Z\. Gong\(2022\)FLTrust: Byzantine\-robust Federated Learning via Trust Bootstrapping\.arXiv preprint arXiv:2012\.13995\(arXiv:2012\.13995\)\.External Links:2012\.13995Cited by:[3rd item](https://arxiv.org/html/2610.12711#S5.I4.i3.p1.1),[TABLE I](https://arxiv.org/html/2610.12711#S5.T1.11.5.1),[TABLE I](https://arxiv.org/html/2610.12711#S5.T1.11.9.1),[TABLE II](https://arxiv.org/html/2610.12711#S5.T2.6.5.1),[TABLE II](https://arxiv.org/html/2610.12711#S5.T2.6.9.1)\.