CauseCollab: Causal Unified and Modality-Agnostic Network for Heterogeneous Collaborative Perception

arXiv cs.AI Papers

Summary

CauseCollab proposes a causal unified and modality-agnostic network to address issues in heterogeneous collaborative perception by disentangling semantic factors from modality-specific confounders, achieving state-of-the-art performance on benchmark datasets.

arXiv:2609.03818v1 Announce Type: new Abstract: Collaborative perception enhances environment understanding through multi-agent information sharing, but its performance in real-world scenarios is constrained by heterogeneous sensor modalities and model architectures. Recent protocol-based two-stage methods alleviate this problem by mapping heterogeneous features into a shared protocol space; however, independently trained modality-specific converters often generate modality-specific pseudo-protocol distributions, leading to semantic inconsistency and error accumulation, which is particularly pronounced in scenarios with large modality discrepancies. To address this issue, we propose CauseCollab, a causal unified and modality-agnostic network. CauseCollab formulates representation learning in the protocol space from a causal perspective, explicitly disentangling semantic factors from modality-specific statistical confounders via causal metric learning. Meanwhile, CauseCollab adopts context-guided Unified Converter for heterogeneous modalities to ensure cross-modal semantic consistency. In addition, integrating new modalities only requires training adapters with minimal parameters. Extensive experiments on the OPV2V and DAIR-V2X datasets demonstrate that CauseCollab achieves state-of-the-art performance, with more significant gains in scenarios involving large modality gaps.
Original Article
View Cached Full Text

Cached at: 09/04/26, 06:12 AM

# CauseCollab: Causal Unified and Modality-Agnostic Network for Heterogeneous Collaborative Perception
Source: [https://arxiv.org/html/2609.03818](https://arxiv.org/html/2609.03818)
Weize LiAffiliation:State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, ChinaYang LiAffiliation:State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, ChinaQuan YuanAffiliation:State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, ChinaCorrespondence to:[yuanquan@bupt\.edu\.cn](mailto:[email protected])Xiaoyuan FuAffiliation:State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, ChinaGuiyang LuoAffiliation:State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, ChinaJinglin LiAffiliation:State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China

###### Abstract

Collaborative perception enhances environment understanding through multi\-agent information sharing, but its performance in real\-world scenarios is constrained by heterogeneous sensor modalities and model architectures\. Recent protocol\-based two\-stage methods alleviate this problem by mapping heterogeneous features into a shared protocol space; however, independently trained modality\-specific converters often generate modality\-specific pseudo\-protocol distributions, leading to semantic inconsistency and error accumulation, which is particularly pronounced in scenarios with large modality discrepancies\. To address this issue, we propose CauseCollab, a causal unified and modality\-agnostic network\. CauseCollab formulates representation learning in the protocol space from a causal perspective, explicitly disentangling semantic factors from modality\-specific statistical confounders via causal metric learning\. Meanwhile, CauseCollab adopts context\-guided Unified Converter for heterogeneous modalities to ensure cross\-modal semantic consistency\. In addition, integrating new modalities only requires training adapters with minimal parameters\. Extensive experiments on the OPV2V and DAIR\-V2X datasets demonstrate that CauseCollab achieves state\-of\-the\-art performance, with more significant gains in scenarios involving large modality gaps\.

###### Keywords:

Machine Learning, Heterogeneous Collaborative Perception, Multi Agent, Causal Inference

††affiliationnotice:Equal contribution## 1Introduction

Collaborative perception shares intermediate features among multiple agents through communication, enabling them to complement each other’s perceptual blind spots, and is widely regarded as a key technique for improving the robustness of autonomous driving systems\([Arnold et al\., 2022](https://arxiv.org/html/2609.03818#bib.bib24);[Han et al\., 2023](https://arxiv.org/html/2609.03818#bib.bib25);[Liu et al\., 2023](https://arxiv.org/html/2609.03818#bib.bib26)\)\. In recent years, extensive research has focused on enhancing the performance of homogeneous collaboration\([Liu et al\., 2020](https://arxiv.org/html/2609.03818#bib.bib27);[Hu et al\., 2022](https://arxiv.org/html/2609.03818#bib.bib13);[Xu et al\., 2022b](https://arxiv.org/html/2609.03818#bib.bib1)\)\. However, in real\-world deployments, collaborative agents often originate from different manufacturers and platforms, exhibiting substantial differences in sensor types, voxel sizes, and network architectures\. It makes intermediate features difficult to fuse and limits the practical effectiveness of collaborative perception\.

To address heterogeneity in collaborative perception, prior work has explored feature transformation modules that map heterogeneous modality features from neighboring agents into the local space of the ego agent\([Zhao et al\., 2023](https://arxiv.org/html/2609.03818#bib.bib35);[Yin et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib9)\)\. Existing approaches can be broadly categorized into two classes\. The first class\([Xu et al\., 2023](https://arxiv.org/html/2609.03818#bib.bib18);[Lu et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib10)\)learns a dedicated mapping for each pair of heterogeneous modalities to achieve one\-stage alignment, but suffers from poor scalability as the training cost grows rapidly with the number of modalities\. The second\([Luo et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib19);[Gao et al\., 2025](https://arxiv.org/html/2609.03818#bib.bib21)\)class introduces a unified protocol modality as an intermediate representation space, and trains a converter and a reconstructor for each modality to map heterogeneous features into this protocol space, achieving improved scalability while maintaining strong performance\.

Although two\-stage heterogeneous adaptation methods based on a protocol space exhibit favorable scalability, their performance is often constrained by error accumulation across stages\. When the converters are trained independently for each modality, each modality tends to learn a locally self\-consistent mapping between its local representation and the protocol representation\. This behavior leads to inconsistent pseudo\-protocol distributions in the protocol space\. As a result, even after all features are projected into a unified protocol space, significant discrepancies in semantic responses and statistical distributions may persist across modalities, which weakens the effectiveness of local semantic reconstruction\.

This issue becomes particularly pronounced when the modality gap is large, where pseudo\-protocol distributions often introduces noisy responses and spurious activations in background regions\. The root cause is that existing methods typically model the protocol space merely as a distribution\-alignment intermediary, failing to explicitly characterize which semantic factors should be shared across modalities and which modality\-specific statistical confounders should be disentangled\.

To address this problem, we propose a unified two\-stage framework from a causal perspective\. Unlike previous methods that treat the protocol space as a reversible intermediate and emphasize distribution matching, our framework focuses on semantic extraction and modal factor disentanglement in Stage\-1, followed by modality\-specific local semantic reconstruction in Stage\-2\.

Specifically, in Stage\-1, we adopt a unified converter composed of a semantic context extractor \(SCE\) and a context\-guided dynamic refiner \(CGDR\) for multi\-modality mixed training\. By taking global semantics as an anchor to constrain local feature refinement, we reduce background\-induced misalignment\. Meanwhile, we construct a structural causal model \(SCM\)\([Schölkopf, 2022](https://arxiv.org/html/2609.03818#bib.bib43)\), depicted in Figure[2](https://arxiv.org/html/2609.03818#S3.F2), to characterize the relationships among heterogeneous modalities in terms of semantic components\. Through a counterfactual intervention strategy based on causal metric learning, we explicitly drive the protocol representations of each modality to approximate the weakly observed protocol modality\. This suppresses pseudo\-protocol shifts induced by residual modal biases and background noise, thus achieving cross\-modal unified semantic alignment\. We design the Mask\-Guided Intervention via SPD Feature Geometry \(MGI\-SPD\) module to enforce causal interventions\. Based on the semantically consistent protocol space, Stage\-2 independently trains a local semantic reconstructor for each modality to restore feature semantics to the local modality\. When a new modality is introduced, we freeze the backbone of the unified converter and only train lightweight adapters to project the new modality into the semantically consistent protocol space, thus achieving expansion at the cost of minimal parameters\.

- •We propose CauseCollab, a causal unified and modality\-agnostic network for heterogeneous collaborative perception\(HCP\)\. It disentangles modality\-specific statistical confounders during the conversion, and realizes the learning of a semantically consistent protocol space based on causal metric learning\. For intermediate features, we design a module called Mask\-Guided Intervention via SPD Feature Geometry\.
- •We design a unified converter composed of a semantic context extractor and a context\-guided dynamic refiner to ensure cross\-modal semantic consistency\. A lightweight adapter is incorporated to enable the adaptation of new modalities at the cost of minimal parameters\.
- •Extensive experiments on the OPV2V and DAIR\-V2X datasets demonstrate that CauseCollab achieves state\-of\-the\-art \(SOTA\) performance among existing heterogeneous collaborative perception methods, with substantial improvements in perception accuracy in scenarios with larger modality gaps\.

## 2Related Work

### 2\.1Collaborative Perception

Collaborative perception allows multiple agents to share and fuse multi\-view information, mitigating the inherent view limitations and occlusions of single\-agent perception\. Intermediate fusion\([Li et al\., 2021](https://arxiv.org/html/2609.03818#bib.bib8);[Xu et al\., 2022a](https://arxiv.org/html/2609.03818#bib.bib34)\)has become the dominant paradigm in collaborative perception, as it preserves rich intermediate semantics while avoiding the high bandwidth requirements and strict synchronization constraints of early fusion\. Methods such as Where2comm\([Hu et al\., 2022](https://arxiv.org/html/2609.03818#bib.bib13)\)and CodeFilling\([Hu et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib14)\)address communication bottlenecks through structured feature compression, while CoAlign\([Lu et al\., 2023](https://arxiv.org/html/2609.03818#bib.bib15)\), CoBEVFlow\([Wei et al\., 2023](https://arxiv.org/html/2609.03818#bib.bib16)\), and CoDiff\([Huang et al\., 2025](https://arxiv.org/html/2609.03818#bib.bib17)\)focus on mitigating feature misalignment caused by localization errors and temporal delays\.

Existing collaborative perception methods excel in homogeneous multi\-agent settings, yet real\-world deployments inevitably require heterogeneous agents\. Recently, several works\([Xiang et al\., 2023](https://arxiv.org/html/2609.03818#bib.bib7);[Shao et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib6)\)have attempted to address heterogeneity in collaborative perception\. MPDA\([Xu et al\., 2023](https://arxiv.org/html/2609.03818#bib.bib18)\)and PnPDA\([Luo et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib19)\)employ one\-stage and two\-stage neighbor\-to\-ego translators to map features into the ego agent’s semantic space\. HEAL\([Lu et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib10)\)performs one\-to\-one mappings via backward alignment, while STAMP\([Gao et al\., 2025](https://arxiv.org/html/2609.03818#bib.bib21)\)introduces a protocol space with reversible mappings between local and protocol representations\. NegoCollab\([Shao et al\., 2025](https://arxiv.org/html/2609.03818#bib.bib20)\)learns a general representation through knowledge distillation\. In contrast to prior approaches, our method aims to eliminate pseudo\-protocol distributions induced by heterogeneous modalities within the protocol space and to learn a semantically consistent unified protocol representation\.

### 2\.2Causal Inference

Causal inference aims to distinguish true causal factors from spurious correlations, characterize causal relationships among variables, and estimate intervention effects that are free from confounding factors\. In recent years, causal inference has been introduced into computer vision tasks\([Wang et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib36);[Li et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib37)\)to improve model robustness\. CIIM\([Yan et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib23)\)explores causally invariant interactions to learn modality\-consistent embeddings\. CWNet\([Zhang et al\., 2025](https://arxiv.org/html/2609.03818#bib.bib22)\)applies causal reasoning to low\-light image enhancement, emphasizing semantic information during the enhancement process\. MSFT\([Qiao et al\., 2025](https://arxiv.org/html/2609.03818#bib.bib12)\)incorporates causal intervention into time\-series forecasting to address confounding effects induced by varying temporal scales\. We introduce structural causal models into collaborative perception and design a novel intervention module operating on intermediate features, enabling more robust cross\-modal semantic alignment\.

## 3Method

![Refer to caption](https://arxiv.org/html/2609.03818v1/camera_ready_structure.png)Figure 1:The overall architecture of CauseCollab\. CauseCollab consists of a modality\-agnostic protocol semantics conversion stage via causal modeling \(Stage\-1\) and a modality\-specific local semantic reconstruction stage \(Stage\-2\)\. Unified Converter incorporates SCE and CGDR modules to derive semantically consistent representations\.### 3\.1Overview Pipeline

The overall architecture of CauseCollab is illustrated in Figure[1](https://arxiv.org/html/2609.03818#S3.F1)\.We focus on the heterogeneous collaborative perception setting, where the ego agentε\\varepsilonreceives observations from multiple collaborative agents𝒱=\{1,…,N\}\\mathcal\{V\}=\\\{1,\\ldots,N\\\}during perception\. Since collaborative agents may belong to different sensor modalities, the modality set is denoted asℳ\\mathcal\{M\}, and directly performing feature fusion suffers from significant modality discrepancies\.

For any collaborative agenti∈𝒱i\\in\\mathcal\{V\}, its raw observation is denoted asximi∈𝒳mix\_\{i\}^\{m\_\{i\}\}\\in\\mathcal\{X\}^\{m\_\{i\}\}, wheremi∈ℳm\_\{i\}\\in\\mathcal\{M\}indicates the sensor modality of agentii\. The ego agent observation is denoted asxεmεx\_\{\\varepsilon\}^\{m\_\{\\varepsilon\}\}\. For each modalitymm, we introduce a modality\-specific encoderEm​\(⋅\)E^\{m\}\(\\cdot\)to map raw inputs into feature representations:

fi=Emi​\(ximi\),fε=Emε​\(xεmε\),f\_\{i\}=E^\{m\_\{i\}\}\(x\_\{i\}^\{m\_\{i\}\}\),\\quad f\_\{\\varepsilon\}=E^\{m\_\{\\varepsilon\}\}\(x\_\{\\varepsilon\}^\{m\_\{\\varepsilon\}\}\),\(1\)wherefi,fε∈ℝC×H×Wf\_\{i\},f\_\{\\varepsilon\}\\in\\mathbb\{R\}^\{C\\times H\\times W\}denote the modality features of collaborative agents and the ego agent, respectively\.

In Stage\-1, we introduce a shared protocol space𝒫\\mathcal\{P\}and project collaborative agent features into this space to obtain unified semantic representations, where each collaborative feature is transformed into its protocol representationpi∈𝒫p\_\{i\}\\in\\mathcal\{P\}\.

In Stage\-2, the ego agent employs a reconstructorRmε​\(⋅\)R^\{m\_\{\\varepsilon\}\}\(\\cdot\)to map protocol features back to local spaceℒ\\mathcal\{L\}, yielding reconstructed collaborative featuresf~i∈ℒ\\tilde\{f\}\_\{i\}\\in\\mathcal\{L\}\. The ego featurefεf\_\{\\varepsilon\}and all reconstructed collaborative features\{f~i\}i=1N\\\{\\tilde\{f\}\_\{i\}\\\}\_\{i=1\}^\{N\}are then fused by a pyramid fusion moduleΦε​\(⋅\)\\Phi\_\{\\varepsilon\}\(\\cdot\)to obtain multi\-scale fused feature:

Fpyramid=Φε​\(fε,f~1,…,f~N\)\.F\_\{\\text\{pyramid\}\}=\\Phi\_\{\\varepsilon\}\(f\_\{\\varepsilon\},\\tilde\{f\}\_\{1\},\\ldots,\\tilde\{f\}\_\{N\}\)\.\(2\)Finally, the fused feature are fed into the task headHε​\(⋅\)H\_\{\\varepsilon\}\(\\cdot\)to produce the final detection output:

z=Hε​\(Fpyramid\)\.z=H\_\{\\varepsilon\}\(F\_\{\\text\{pyramid\}\}\)\.\(3\)

### 3\.2Stage\-1: Protocol Semantic Conversion via Causal Modeling

##### Structural Causal Modeling\.

To distinguish modality\-invariant semantic factors from modality\-specific statistical confounders that should be suppressed, we introduce Structural Causal Modeling \(SCM\) in Stage\-1 to explicitly characterize the causal generation process of representations\. As illustrated in Figure[2](https://arxiv.org/html/2609.03818#S3.F2), the variables include the protocol\-modality representationPP, the heterogeneous modality representationFmF\_\{m\}, the protocol\-space heterogeneous representationFm→pF\_\{m\\rightarrow p\}, the locally reconstructed representationFm→p→mF\_\{m\\rightarrow p\\rightarrow m\}, the true semantic factorSS, the protocol\-modality statistical noiseNpN\_\{p\}, and the heterogeneous\-modality statistical noiseNmN\_\{m\}\.

Without proper constraints, the statistical noisesNpN\_\{p\}andNmN\_\{m\}can affectFmF\_\{m\}andPPthrough the encoder, thereby inducing spurious correlation paths in the protocol space that are irrelevant to the true semanticsSS\. Consequently, the model may exploit modality\-style differences as discriminative cues, leading to modality\-specific spurious semantic clusters inFm→pF\_\{m\\rightarrow p\}\. To obtain a semantically consistent protocol space, we naturally regardSSas the causal factor, while treatingNpN\_\{p\}andNmN\_\{m\}as confounders\.

Based on the above SCM, the learning objective of Stage\-1 can be stated as follows: while maintaining high sensitivity ofFm→pF\_\{m\\rightarrow p\}to the semantic factorSS, we reduce its dependence on modality statistical noisesNpN\_\{p\}andNmN\_\{m\}\. SinceSSis not directly observable, we use the protocol\-modality representationPPas a weak observation during training, and introduce SPD mask\-guided intervention together with causal metric learning to achieve the above causal objective\.

Figure 2:Representation\-level Structural Causal Model\(SCM\) for Two\-Stage Heterogeneous Collaborative Perception\. Black edges denote causal paths, while red edges denote spurious correlation paths\. Solid edges represent direct effects, and dashed edges indicate indirect effects on the protocol\-space representation\.
##### Mask\-Guided Intervention via SPD Feature Geometry\.

After heterogeneous features are transformed into the protocol space, they still retain significant modality\-specific statistical shifts, which are mainly reflected in the discrepancies of first\-order means and second\-order correlation structures of channel distributions\([Huang and Belongie, 2017](https://arxiv.org/html/2609.03818#bib.bib40);[Li et al\., 2017](https://arxiv.org/html/2609.03818#bib.bib41)\)\. To address this issue, we design Mask\-Guided Intervention via SPD Feature Geometry, and realize the separation of confounders through causal metric learning\. Specifically, we regard Bird’s\-Eye View \(BEV\) features as a collection of high\-dimensional random vectors sampled from various spatial locations, and construct closed\-form statistical mappings on the geometry of symmetric positive definite \(SPD\) covariance matrices\([Huang and Gool, 2017](https://arxiv.org/html/2609.03818#bib.bib42)\); meanwhile, we introduce semantic mask\-guided spatially adaptive strength modulation to perform differentiated statistical suppression and statistical injection on foreground and background regions, thereby implementing counterfactual intervention onNpN\_\{p\}andNmN\_\{m\}without disrupting the feature semantic factorSS\.

Given a protocol feature mappi∈ℝC×H×Wp\_\{i\}\\in\\mathbb\{R\}^\{C\\times H\\times W\}, we reshape it intoXi∈ℝC×NX\_\{i\}\\in\\mathbb\{R\}^\{C\\times N\}withN=H×WN=H\\times W, where each column vectorxn∈ℝCx\_\{n\}\\in\\mathbb\{R\}^\{C\}represents the channel feature at the corresponding spatial location\. Under this representation, modality style is modeled by the first\-order and second\-order statistics\(μ,Σ\)\(\\mu,\\Sigma\)of the feature setXX\.

To characterize the discrepancy in semantic contributions between foreground and background regions, we introduce a semantic maskmi∈\[0,1\]1×Nm\_\{i\}\\in\[0,1\]^\{1\\times N\}generated by inputting heterogeneous features into a mask generatorg⁡\(⋅\)g\(\\cdot\), and construct non\-negative weightswwbased on this mask to define weighted statistics:

μ=∑n=1Nwn​xn∑n=1Nwn,\\mu=\\frac\{\\sum\_\{n=1\}^\{N\}w\_\{n\}x\_\{n\}\}\{\\sum\_\{n=1\}^\{N\}w\_\{n\}\},\(4\)Σ=∑n=1Nwn​\(xn−μ\)​\(xn−μ\)⊤∑n=1Nwn\+ε​I,\\Sigma=\\frac\{\\sum\_\{n=1\}^\{N\}w\_\{n\}\(x\_\{n\}\-\\mu\)\(x\_\{n\}\-\\mu\)^\{\\top\}\}\{\\sum\_\{n=1\}^\{N\}w\_\{n\}\}\+\\varepsilon I,\(5\)whereΣ∈𝒮\+C\\Sigma\\in\\mathcal\{S\}\_\{\+\}^\{C\}lies on the SPD geometry, andε​I\\varepsilon Iis added to ensure numerical stability\.

On the SPD geometry, we adopt a whitening–coloring linear mapping to realize a closed\-form transformation from source statistics\(μs,Σs\)\(\\mu\_\{s\},\\Sigma\_\{s\}\)to target statistics\(μt,Σt\)\(\\mu\_\{t\},\\Sigma\_\{t\}\):

Ts→t​\(X\)=μt\+Σt12​Σs−12​\(X−μs\),T\_\{s\\to t\}\(X\)=\\mu\_\{t\}\+\\Sigma\_\{t\}^\{\\frac\{1\}\{2\}\}\\Sigma\_\{s\}^\{\-\\frac\{1\}\{2\}\}\(X\-\\mu\_\{s\}\),\(6\)whereΣ1/2\\Sigma^\{1/2\}andΣ−1/2\\Sigma^\{\-1/2\}are computed via eigendecomposition of SPD matrices to ensure numerical stability and preserve positive definiteness\. The mapping performs a linear adjustment of the feature mean and correlation structure along the channel dimension only, without altering the spatial structure of the feature map, thereby guaranteeing that the BEV semantics are not compromised\.

Under the above formulation, we apply counterfactual intervention to the protocol featurefpf\_\{p\}and construct a pair of positive and negative samples for causal metric learning: the demodalized protocol featurepi\+p\_\{i\}^\{\+\}serves as the positive sample, and the confounded protocol featurepi−p\_\{i\}^\{\-\}acts as the negative sample\. Specifically,pi\+p\_\{i\}^\{\+\}is obtained by mapping the channel statistics\(μc,Σc\)\(\\mu\_\{c\},\\Sigma\_\{c\}\)offpf\_\{p\}to a neutral target distribution\(μ0,Σ0\)\(\\mu\_\{0\},\\Sigma\_\{0\}\)via whitening coloring transformation combined with semantic mask\-guided spatial interpolation, thereby suppressing modality\-specific statistics while preserving the spatial semantic structure; in contrast, after completing the aforementioned suppression,pi−p\_\{i\}^\{\-\}is generated by introducing controlled injection of heterogeneous modality statistics\(μh,Σh\)\(\\mu\_\{h\},\\Sigma\_\{h\}\)through isomorphic statistical mapping and spatial modulation, such that modality\-induced statistical perturbations are mixed in while maintaining semantic consistency\. Through this intervention construction based on SPD statistical geometry,pi\+p\_\{i\}^\{\+\}provides a demodalized semantic positive sample, whilepi−p\_\{i\}^\{\-\}acts as a hard negative sample with controlled modality shifts introduced, jointly driving the unified converter to learn modality\-agnostic protocol representations\. It is worth noting that MGI\-SPD is used only during Stage\-1 training to remove modality\-specific confounding factors; it is not involved during inference\. More details can be found in Appendix[B\.1](https://arxiv.org/html/2609.03818#A2.SS1)\.

##### Semantic Context Extractor\.

To learn representations with stronger semantic consistency in the protocol space, we design a semantic context extractorG⁡\(⋅\)G\(\\cdot\)for extracting high\-level semantic information\. This module takes the modality featurefi∈ℝC×H×Wf\_\{i\}\\in\\mathbb\{R\}^\{C\\times H\\times W\}generated by the heterogeneous modality encoders as input, and outputs the global semantic context featureci∈ℝCg×Hg×Wgc\_\{i\}\\in\\mathbb\{R\}^\{C\_\{g\}\\times H\_\{g\}\\times W\_\{g\}\}\.

G⁡\(⋅\)G\(\\cdot\)adopts a hierarchical structure, and gradually enhances the global semantic modeling capability through two\-stage downsampling and semantic extraction\. Specifically, we construct two levels of semantic representations:

x\(1\)=ℬ1​\(𝒟1​\(fi\)\),x\(2\)=ℬ2​\(𝒟2​\(x\(1\)\)\),x^\{\(1\)\}=\\mathcal\{B\}\_\{1\}\(\\mathcal\{D\}\_\{1\}\(f\_\{i\}\)\),\\quad x^\{\(2\)\}=\\mathcal\{B\}\_\{2\}\(\\mathcal\{D\}\_\{2\}\(x^\{\(1\)\}\)\),\(7\)and fuse the two\-level features to obtain the context representation:

ci=Fuse⁡\(x\(1\),x\(2\)\)\.c\_\{i\}=\\mathrm\{Fuse\}\(x^\{\(1\)\},x^\{\(2\)\}\)\.\(8\)
where𝒟1,𝒟2\\mathcal\{D\}\_\{1\},\\mathcal\{D\}\_\{2\}denote downsampling convolutions,ℬ1,ℬ2\\mathcal\{B\}\_\{1\},\\mathcal\{B\}\_\{2\}are semantic extraction blocks, andFuse⁡\(⋅\)\\mathrm\{Fuse\}\(\\cdot\)is the feature fusion operator\. Specifically,Fuse⁡\(⋅\)\\mathrm\{Fuse\}\(\\cdot\)first aligns the channel dimensions using1×11\\times 1convolutions, performs spatial alignment via interpolation, and then applies weighted concatenation followed by a depth\-wise3×33\\times 3convolution for spatial mixing\. Finally, we apply channel recalibration, implemented by the classical Squeeze\-and\-Excitation module, and normalization tocic\_\{i\}at the output stage to enhance task\-relevant semantic responses and stabilize the feature distribution, thereby providing reliable global semantic context for subsequent dynamic refinement\.

##### Context\-Guided Dynamic Refiner\.

In Stage\-1, we introduce the Context\-Guided Dynamic RefinerT⁡\(⋅\)T\(\\cdot\)\. This module performs layer\-wise semantic correction on local modality features under semantic context constraints, thereby generating protocol\-space representations with stronger semantic consistency\.

Given the local modality featurefi∈ℝC×H×Wf\_\{i\}\\in\\mathbb\{R\}^\{C\\times H\\times W\}of theii\-th agent and the semantic context featurecic\_\{i\}extracted by the Semantic Context Extractor \(SCE\), we first align the context feature in both channel and spatial dimensions to obtain the conditional context representation:

c^i=U⁡\(ϕ⁡\(ci\)\),\\hat\{c\}\_\{i\}=U\\bigl\(\\phi\(c\_\{i\}\)\\bigr\),\(9\)whereϕ⁡\(⋅\)\\phi\(\\cdot\)denotes a1×11\\times 1convolution andU⁡\(⋅\)U\(\\cdot\)represents linear interpolation\. Subsequently, the refiner takes\(fi,c^i\)\(f\_\{i\},\\hat\{c\}\_\{i\}\)as joint input and progressively generates protocol\-space features through a stack of context\-conditioned transformations:

pi=T⁡\(fi,c^i\)\.p\_\{i\}=T\(f\_\{i\},\\hat\{c\}\_\{i\}\)\.\(10\)
In practice, the refiner is composed of multiple dynamic refinement blocks\. For theℓ\\ell\-th layer, the update of local modality features is conditioned on the aligned context representation, which is formulated as:

fi\(ℓ\+1\)=Tℓ​\(fi\(ℓ\),c^i\)\.f\_\{i\}^\{\(\\ell\+1\)\}=T\_\{\\ell\}\\bigl\(f\_\{i\}^\{\(\\ell\)\},\\hat\{c\}\_\{i\}\\bigr\)\.\(11\)
Within each dynamic refinement block, local features are first modulated by FiLM\-style multi\-head feature modulation under the guidance of semantic context, and then filtered by a gating functiongate⁡\(⋅\)\\mathrm\{gate\}\(\\cdot\), implemented as a1×11\\times 1convolution followed by SiLU activation, to suppress non\-informative regions and noise responses, thereby stabilizing the feature transformation process\.

By continuously enforcing semantic context constraints throughout the transformation, the resulting protocol representationpip\_\{i\}not only preserves local discriminative details but also anchors to consistent global semantics, providing reliable inputs for local semantic reconstruction and task prediction in Stage\-2\.

##### Training Strategy\.

Table 1:Performance comparison on the OPV2V dataset using AP@30/AP@50, where the ego agent isLpp4L\_\{\\mathrm\{pp4\}\}\.ALLdenotes the scenario with all collaborative agents\.MethodAP@30↑\\uparrowAP@50↑\\uparrow\+CEff\+\\,C\_\{\\mathrm\{Eff\}\}\+Ls​d​1\+\\,L\_\{sd1\}\+CRes\+\\,C\_\{\\mathrm\{Res\}\}ALL\\mathrm\{ALL\}\+CEff\+\\,C\_\{\\mathrm\{Eff\}\}\+Ls​d​1\+\\,L\_\{sd1\}\+CRes\+\\,C\_\{\\mathrm\{Res\}\}ALL\\mathrm\{ALL\}MPDA0\.9470\.9670\.9670\.9690\.9390\.9620\.9620\.966PnPDA0\.9360\.9650\.9660\.9700\.9310\.9620\.9630\.967HEAL0\.9410\.9580\.9590\.9630\.9350\.9540\.9550\.960STAMP0\.9450\.9670\.9680\.9720\.9400\.9640\.9650\.969CauseCollab0\.9520\.9720\.9730\.9780\.9440\.9680\.9690\.975Stage\-1 aims to learn semantically consistent protocol representations: this stage enhances the unified semantic factorSSacross modalities while suppressing the interference of modality\-specific statistical noiseNpN\_\{p\}andNmN\_\{m\}, thereby reducing pseudo\-protocol shifts and erroneous activations in background regions\.

The overall objective function of Stage\-1 is defined as follows:

ℒStage\-1\\displaystyle\\mathcal\{L\}\_\{\\text\{Stage\-1\}\}=λcon​ℒcontrast\+λctx​\(ℒctx\-ssim\+ℒctx\-cos\)\\displaystyle=\\lambda\_\{\\text\{con\}\}\\mathcal\{L\}\_\{\\text\{contrast\}\}\+\\lambda\_\{\\text\{ctx\}\}\\bigl\(\\mathcal\{L\}\_\{\\text\{ctx\-ssim\}\}\+\\mathcal\{L\}\_\{\\text\{ctx\-cos\}\}\\bigr\)\(12\)\+λssim​ℒssim\+λtask\(1\)​ℒtaskp,\\displaystyle\+\\lambda\_\{\\text\{ssim\}\}\\mathcal\{L\}\_\{\\text\{ssim\}\}\+\\lambda^\{\(1\)\}\_\{\\text\{task\}\}\\mathcal\{L\}\_\{\\text\{task\}\}^\{p\},whereλcon\\lambda\_\{\\text\{con\}\},λctx\\lambda\_\{\\text\{ctx\}\},λssim\\lambda\_\{\\text\{ssim\}\}andλtask\(1\)\\lambda^\{\(1\)\}\_\{\\text\{task\}\}are weighting coefficients\.

Among these loss terms,ℒcontrast\\mathcal\{L\}\_\{\\text\{contrast\}\}is an InfoNCE\-style contrastive loss\([van den Oord et al\., 2019](https://arxiv.org/html/2609.03818#bib.bib3)\)for causal metric learning, wherepip\_\{i\}is used as the anchor, the demodalized intervention featurepi\+p\_\{i\}^\{\+\}serves as the positive sample, and the confounded intervention featurepi−p\_\{i\}^\{\-\}serves as the negative sample;ℒctx\-ssim\\mathcal\{L\}\_\{\\text\{ctx\-ssim\}\}andℒctx\-cos\\mathcal\{L\}\_\{\\text\{ctx\-cos\}\}are semantic context consistency constraints applied between the semantic context featurecic\_\{i\}and the downsampled positive intervention featurepi\+p\_\{i\}^\{\+\}, aligning high\-dimensional semantics from the perspectives of spatial structure and channel response, respectively;ℒssim\\mathcal\{L\}\_\{\\text\{ssim\}\}enforces semantic consistency between the protocol\-modality featurefpf\_\{p\}and the converted protocol representationpip\_\{i\}, constraining dynamic refinement to achieve protocol mapping while preserving spatial semantic structure;ℒtaskp\\mathcal\{L\}\_\{\\text\{task\}\}^\{p\}is the protocol\-head detection loss, implemented as the focal loss plus the smoothℒ1\\mathcal\{L\}\_\{1\}loss, ensuring protocol representations have both semantic consistency and sufficient discriminability for downstream detection tasks\.

### 3\.3Stage\-2: Local Semantic Reconstruction

After Stage\-1, features from different modalities are mapped into a semantically consistent protocol space\. Since heterogeneous modalities differ in resolution and representation characteristics, Stage\-2 reconstructs protocol back to each modality\-specific local space to fit modality\-specific heads\.

With the unified converter in Stage\-1 frozen, we train a modality\-specific local reconstructorRm​\(⋅\)R^\{m\}\(\\cdot\)for each modalitymmto map features back to the local feature space:

f~i=Rm​\(pi\),\\tilde\{f\}\_\{i\}=R^\{m\}\(p\_\{i\}\),\(13\)wherepip\_\{i\}denotes the protocol\-space feature produced by Stage\-1\. The reconstructor is designed following the ConvNeXt\([Liu et al\., 2022](https://arxiv.org/html/2609.03818#bib.bib5)\)architecture to ensure stable optimization\.

For modalitymm, Stage\-2 is optimized with task supervision and consistency constraints\. The overall objective is:

ℒStage\-2=λtask\(2\)​ℒtaskϵ\+λmse​ℒm​s​e\+λcons​ℒc​o​n​s,\\mathcal\{L\}\_\{\\text\{Stage\-2\}\}=\\lambda^\{\(2\)\}\_\{\\text\{task\}\}\\mathcal\{L\}^\{\\epsilon\}\_\{\\text\{task\}\}\+\\lambda\_\{\\text\{mse\}\}\\mathcal\{L\}\_\{mse\}\+\\lambda\_\{\\text\{cons\}\}\\mathcal\{L\}\_\{cons\},\(14\)
whereλtask\(2\)\\lambda^\{\(2\)\}\_\{\\text\{task\}\},λmse\\lambda\_\{\\text\{mse\}\}, andλcons\\lambda\_\{\\text\{cons\}\}are weighting coefficients\.ℒtaskϵ\\mathcal\{L\}^\{\\epsilon\}\_\{\\text\{task\}\}is the modality\-specific task loss for downstream task supervision;ℒm​s​e\\mathcal\{L\}\_\{mse\}is the MSE loss between reconstructed and original local features;ℒc​o​n​s\\mathcal\{L\}\_\{cons\}is the consistency loss to stabilize the mapping between protocol and local feature spaces\.

### 3\.4Generalization

The initial collaborative agents have learned modality\-agnostic causal semantics with consistent representations in the protocol space\. For a new modality, we freeze the backbone of the unified converter in Stage\-1 and only employ lightweight adapters to align the new modality to the established protocol\-consistent space; in Stage\-2, we further train a local semantic reconstructor to recover local space\.

The lightweight adaptation in Stage\-1 is achieved by inserting a small number of bottleneck adapters at key locations in the unified converter, while keeping the backbone parameters frozen\. The adapters are placed after the downsampling operations in the SCE and between modules of the CGDR, so that the new modality is progressively aligned to modality\-invariant semantics through scale transitions and fine\-grained conversions\. Each adapter follows a low\-rank bottleneck residual design and is initialized with a scaling coefficientα=0\\alpha=0, enabling effective compensation for the new modality with negligible parameter overhead while preserving the original protocol semantic consistency\.

## 4Experiments

Table 2:Experiments of different methods on the OPV2V dataset under the setting of larger domain gap, along with the parameters required for new modality adaptation\. The notationA\+BA\+Bindicates that the ego agent is configured with modalityAA, while all neighbor agents are configured with modalityBB\. Model parameters are measured in megabytes \(MB\)\.MethodAP@30/AP@50↑\\uparrow\#Parmas\(New Modality\)Lp​p​4\+CEffL\_\{pp4\}\+C\_\{\\mathrm\{Eff\}\}CEff\+CResC\_\{\\mathrm\{Eff\}\}\+C\_\{\\mathrm\{Res\}\}Lp​p​4\+Ls​d​2newL\_\{pp4\}\+L\_\{sd2^\{\\text\{new\}\}\}Ls​d​2new\+CEffB1newL\_\{sd2^\{\\text\{new\}\}\}\+C\_\{\\mathrm\{EffB1\}^\{\\text\{new\}\}\}MPDA0\.967 / 0\.9610\.711 / 0\.5970\.949 / 0\.9430\.941 / 0\.9375\.00PnPDA0\.966 / 0\.9620\.654 / 0\.5970\.980 / 0\.9790\.951 / 0\.9464\.37HEAL0\.960 / 0\.9560\.627 / 0\.5500\.980 / 0\.9780\.936 / 0\.93217\.10STAMP0\.961 / 0\.9570\.730 / 0\.6340\.981 /0\.9800\.958 / 0\.9521\.18CauseCollab0\.971/0\.9660\.772/0\.6700\.983/0\.9800\.966/0\.9590\.68Table 3:Performance comparison of heterogeneous collaboration on real\-world datasets DAIR\-V2X\.MethodAP@30/AP@50↑\\uparrowLpp4\+CEffL\_\{\\mathrm\{pp4\}\}\+C\_\{\\mathrm\{Eff\}\}CEff\+Lpp4C\_\{\\mathrm\{Eff\}\}\+L\_\{\\mathrm\{pp4\}\}MPDA0\.792 / 0\.6850\.405 / 0\.228PnPDA0\.790 /0\.7380\.408 / 0\.235HEAL0\.738 / 0\.6920\.410 / 0\.253STAMP0\.782 / 0\.7310\.394 / 0\.249CauseCollab0\.795/ 0\.7330\.647/0\.464### 4\.1Settings

##### Datasets\.

We conduct experiments on OPV2V\([Xu et al\., 2022c](https://arxiv.org/html/2609.03818#bib.bib31)\)and DAIR\-V2X\([Yu et al\., 2022](https://arxiv.org/html/2609.03818#bib.bib32)\)\. OPV2V is a simulator\-collected multi\-agent dataset tailored for heterogeneous collaborative perception\. DAIR\-V2X is a real\-world V2X dataset collected in traffic environments, consisting of both agent\-side and roadside LiDAR and camera data\.

##### Experimental Details\.

We adopt heterogeneous feature encoders spanning different sensor types and network architectures\. For LiDAR, we use PointPillar\([Lang et al\., 2019](https://arxiv.org/html/2609.03818#bib.bib28)\)and SECOND\([Yan et al\., 2018](https://arxiv.org/html/2609.03818#bib.bib11)\)as two representative encoders\. For cameras, we use an image encoder based on the Lift\-Splat\-Shoot \(LSS\)\([Philion and Fidler, 2020](https://arxiv.org/html/2609.03818#bib.bib33)\)framework, which employs ResNet\([He et al\., 2015](https://arxiv.org/html/2609.03818#bib.bib29)\)and EfficientNet\([Tan and Le, 2019](https://arxiv.org/html/2609.03818#bib.bib30)\)as the backbone networks\. Specifically, we construct four heterogeneous modality configurations, denoted asLpp4L\_\{\\mathrm\{pp4\}\},CEffC\_\{\\mathrm\{Eff\}\},Lsd1L\_\{\\mathrm\{sd1\}\}, andCResC\_\{\\mathrm\{Res\}\}\. These configurations differ in sensor types, voxel size, and backbone design\. Detailed settings are provided in Table[6](https://arxiv.org/html/2609.03818#A1.T6)\.

We employ a two\-stage training scheme\. In Stage\-1, features extracted by the four heterogeneous encoders are mixed and fed into shared converter\. The converter is trained with multi\-modal mixed batches to align heterogeneous modality spaces into a unified protocol space\. In Stage\-2, we train a modality\-specific local semantic reconstructor for each modality to recover collaborator features in its local representation space\. When integrating a new modality, we freeze the shared converter backbone and only perform lightweight adaptation, together with training the local reconstructor for the new modality\. Notably, during adaptation, the encoders, fusion modules, and classification heads of previously deployed modalities remain frozen and inaccessible, which satisfies practical deployment requirements on privacy preservation and non\-modifiability of existing modalities\.

### 4\.2Performance Comparison

##### Collaboration on OPV2V\.

We evaluate our method on the 3D object detection task and compare it with prior approaches\. We select two one\-stage methods, MPDA\([Xu et al\., 2023](https://arxiv.org/html/2609.03818#bib.bib18)\)and HEAL\([Lu et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib10)\), and two two\-stage methods, PnPDA\([Luo et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib19)\)and STAMP\([Gao et al\., 2025](https://arxiv.org/html/2609.03818#bib.bib21)\)\. For a fair comparison, we adopt the same pyramid fusion strategy used in prior works\([Lu et al\., 2024](https://arxiv.org/html/2609.03818#bib.bib10);[Gao et al\., 2025](https://arxiv.org/html/2609.03818#bib.bib21)\)\. Table[1](https://arxiv.org/html/2609.03818#S3.T1)reports the results on OPV2V under a general heterogeneous collaboration setting\. We set the ego agent to theLpp4L\_\{\\text\{pp4\}\}modality and progressively add three heterogeneous\-modality neighbors for collaboration\. CauseCollab achieves the best performance under both AP@0\.3 and AP@0\.5 metrics after each heterogeneous modality agent is progressively added\.

Figure 3:Robustness comparison of different methods under pose noise\. We evaluate the AP@0\.3 and AP@0\.5 performance as the pose noise standard deviation increases from 0\.0 to 0\.5\.Furthermore, we evaluate the robustness of the model against localization errors by injecting Gaussian noise with standard deviationσ\\sigmaranging from 0\.0 to 0\.5 into agent poses, and the experimental results are shown in Figure[3](https://arxiv.org/html/2609.03818#S4.F3)\. The results demonstrate that the performance of CauseCollab is better than or comparable to that of other methods at most noise levels\.

To validate semantic consistency in the protocol space, we set the ego agent to one modality and neighbor agents to a single different modality, which emulates a larger modality gap for heterogeneous collaboration\. We then evaluate the model’s robustness in this challenging scenario, with results provided in Table[2](https://arxiv.org/html/2609.03818#S4.T2)\. When the ego isCEffC\_\{\\text\{Eff\}\}and the neighbor isCResC\_\{\\text\{Res\}\}, our method improves AP by 4\.2% at IoU=0\.3 and by 3\.6% at IoU=0\.5 compared to four other methods\. Similarly, when the ego isLpp4L\_\{\\text\{pp4\}\}and the neighbor isCEffC\_\{\\text\{Eff\}\}, the AP gains are 0\.5% and 0\.4% under the two thresholds, with improvements even for the higher\-performance combination\. These results confirm that our method preserves more semantically consistent information during collaboration\.

##### Collaboration on DAIR\-V2X\.

We further conduct performance comparisons on the real\-world DAIR\-V2X dataset\. Since DAIR\-V2X only provides two agent perspectives for the same scene, we set the Ego agent to theLp​p​4L\_\{pp4\}andCEffC\_\{\\mathrm\{Eff\}\}modalities respectively for collaboration\. As shown in Table[3](https://arxiv.org/html/2609.03818#S4.T3), our method still outperforms all previous methods overall\.

##### Scalability\.

Heterogeneous collaboration methods should be scalable to incorporate novel modalities\. Starting from a unified converter trained on the four heterogeneous modalities above, we adapt it to two novel modalities,Lsd2L\_\{\\text\{sd2\}\}andCEffB1\\text\{C\}\_\{\\text\{EffB1\}\}, via fine\-tuning\. We then use the novel modality as the neighbor agent for collaboration, as shown in Table[2](https://arxiv.org/html/2609.03818#S4.T2)\. We observe that, for novel modalities, our method achieves higher average precision with fewer trainable parameters, indicating that the unified converter enables semantic consistency and parameter reuse\.

Table 4:Ablation study: Evaluating the impact of individual components on the performance of CauseCollab\. Experiments are conducted under the modality configuration ofLpp4\+CEff\+Lsd1\+CResL\_\{\\mathrm\{pp4\}\}\+C\_\{\\mathrm\{Eff\}\}\+L\_\{\\mathrm\{sd1\}\}\+C\_\{\\mathrm\{Res\}\}\.AP@30↑\\uparrowAP@50↑\\uparrowCauseCollab0\.9780\.975\-w/o Causal Intervention0\.9690\.965\-w/o Mask Generator0\.9720\.970\-w/o Semantic Context0\.8370\.834\-w/o CGDR0\.9720\.969\-w/o SCE \+ CGDR0\.9680\.966\-w/o Lightweight Adapters0\.9090\.904

### 4\.3Ablation Study

We conducted a comprehensive ablation study on the OPV2V dataset\. In Stage\-1, we ablate each component of the model individually and evaluate the heterogeneous collaboration performance under the general collaboration setting\. As shown in Table[4](https://arxiv.org/html/2609.03818#S4.T4), removing any single component consistently leads to performance degradation\.

Specifically, after removing the causal intervention module, the AP metrics at IoU thresholds of 0\.3 and 0\.5 decrease significantly, respectively, highlighting the critical role of our causal intervention in improving the semantic consistency of the protocol space\. In addition, removing the intervention mask also causes performance degradation, verifying the effectiveness of the mask\-guided intervention mechanism\.

We directly remove the semantic context guidance, which results in a substantial performance drop, demonstrating the necessity of semantic context for the unified converter\. Replacing CGDR and SCE \+ CGDR with a single\-tower ConvNeXt of the same number of layers, respectively, leads to additional performance degradation, and this downward trend demonstrates the necessity of the coexistence of SCE and CGDR\. Finally, removing the adapter for new modalities causes performance degradation, which demonstrates the value of adapters in incorporating new modalities\.

### 4\.4Qualitative Evaluation

![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/feat/m4_cav_2_crop.png)\(a\)Lpp4L\_\{\\text\{pp4\}\}Feature
![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/feat/m1_cav_2_crop.png)\(b\)CEffC\_\{\\mathrm\{Eff\}\}Feature
![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/feat/m1_to_protocol_cav_2_crop.png)\(c\)CEffC\_\{\\mathrm\{Eff\}\}Feature in Protocol Space \(STAMP\)
![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/feat/cbc_after_cnet_m1_crop.png)\(d\)CEffC\_\{\\mathrm\{Eff\}\}Feature in Protocol Space \(CauseCollab\)

Figure 4:Visualization of intermediate features before and after conversion\.To provide more intuitive evidence that CauseCollab learns a semantically consistent protocol space for heterogeneous collaboration, we visualize the intermediate features of LiDAR and camera modalities for the same scene in Figure[4](https://arxiv.org/html/2609.03818#S4.F4), as well as the protocol space features converted from the camera modality by STAMP and our method, respectively\. As shown in Figure[4](https://arxiv.org/html/2609.03818#S4.F4)\(a\) and \(b\), a significant semantic gap exists between the two modalities’ intermediate features\. STAMP introduces semantically irrelevant noise in background regions \(Figure[4](https://arxiv.org/html/2609.03818#S4.F4)\(c\)\), whereas our method achieves cleaner and more effective protocol semantics conversion, better preserving cross\-modality semantic consistency \(Figure[4](https://arxiv.org/html/2609.03818#S4.F4)\(d\)\)\.

## 5Conclusion

In this paper, we propose a causal unified and modality\-agnostic network called CauseCollab, to address the issues of semantic inconsistency and modality confounding in the protocol space of heterogeneous collaborative perception\. CauseCollab achieves modality\-agnostic consistent semantics via causal metric learning, and enables the extension of new modalities to the semantically consistent protocol space by training only lightweight adapters with minimal parameters\. Extensive experiments demonstrate the effectiveness of our proposed method\.

## Acknowledgements

This work was supported in part by the Natural Science Foundation of China under Grant 62272053 and Grant 62472048, in part by the Beijing Nova Program under Grant 20230484364, in part by Beijing Natural Science Foundation under Grant L242081, and in part by BUPT Excellent Ph\.D\. Students Foundation\.

## Impact Statement

This work advances heterogeneous collaborative perception by addressing semantic inconsistency and modality confounding through a causally unified, modality\-agnostic framework\. CauseCollab enables seamless integration of new sensing modalities with minimal computational overhead, accelerating the deployment of multi\-modal collaborative systems in autonomous driving and intelligent transportation\. Potential risks may arise from safety\-critical deployment and privacy concerns in multi\-agent perception systems\. However, our framework mitigates these risks to some extent by improving cross\-modal semantic consistency and enabling adaptation without accessing or modifying previously deployed modality\-specific encoders, fusion modules, and detection heads\.

## References

- E\. Arnold, M\. Dianati, R\. d\. Temple, and S\. FallahCooperative perception for 3d object detection in driving scenarios using infrastructure sensors\.23\(3\),pp\. 1852–1864\.External Links:[Document](https://dx.doi.org/10.1109/TITS.2020.3028424),1912\.12147,[Link](http://arxiv.org/abs/1912.12147)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p1.1)\.
- Dosovitskiyet al\.\(2017\)A\. Dosovitskiy, G\. Ros, F\. Codevilla, A\. Lopez, and V\. KoltunCARLA: an open urban driving simulator\.External Links:1711\.03938,[Link](https://arxiv.org/abs/1711.03938)Cited by:[§A\.1](https://arxiv.org/html/2609.03818#A1.SS1.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2025\)X\. Gao, R\. Xu, J\. Li, Z\. Wang, Z\. Fan, and Z\. TuSTAMP: scalable task and model\-agnostic collaborative perception\.InProceedings of the International Conference on Learning Representations \(ICLR\),External Links:2501\.18616,[Link](https://arxiv.org/abs/2501.18616)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p2.1),[§4\.2](https://arxiv.org/html/2609.03818#S4.SS2.SSS0.Px1.p1.1)\.
- Hanet al\.\(2023\)Y\. Han, H\. Zhang, H\. Li, Y\. Jin, C\. Lang, and Y\. LiCollaborative perception in autonomous driving: methods, datasets and challenges\.IEEE Intelligent Transportation Systems Magazine15\(6\),pp\. 131–151\.External Links:[Document](https://dx.doi.org/10.1109/MITS.2023.3298534),[Link](https://arxiv.org/abs/2301.06262),2301\.06262Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p1.1)\.
- Heet al\.\(2015\)K\. He, X\. Zhang, S\. Ren, and J\. SunDeep residual learning for image recognition\.External Links:1512\.03385,[Link](https://arxiv.org/abs/1512.03385)Cited by:[§A\.3](https://arxiv.org/html/2609.03818#A1.SS3.p1.1),[§4\.1](https://arxiv.org/html/2609.03818#S4.SS1.SSS0.Px2.p1.1)\.
- Huet al\.\(2019\)J\. Hu, L\. Shen, S\. Albanie, G\. Sun, and E\. WuSqueeze\-and\-excitation networks\.IEEE\.Note:CVPR 2019External Links:1709\.01507,[Link](https://arxiv.org/abs/1709.01507)Cited by:[§B\.3](https://arxiv.org/html/2609.03818#A2.SS3.p2.1)\.
- Huet al\.\(2022\)Y\. Hu, S\. Fang, Z\. Lei, Y\. Zhong, and S\. ChenWhere2comm: communication\-efficient collaborative perception via spatial confidence maps\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2209\.12836,[Link](https://arxiv.org/abs/2209.12836)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p1.1)\.
- Huet al\.\(2024\)Y\. Hu, J\. Peng, S\. Liu, J\. Ge, S\. Liu, and S\. ChenCommunication\-efficient collaborative perception via information filling with codebook\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:2405\.04966,[Link](https://arxiv.org/abs/2405.04966)Cited by:[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p1.1)\.
- Huang and Belongie \(2017\)X\. Huang and S\. BelongieArbitrary style transfer in real\-time with adaptive instance normalization\.InProceedings of the IEEE International Conference on Computer Vision \(ICCV\),External Links:1703\.06868,[Link](https://arxiv.org/abs/1703.06868)Cited by:[§3\.2](https://arxiv.org/html/2609.03818#S3.SS2.SSS0.Px2.p1.1)\.
- Huanget al\.\(2025\)Z\. Huang, S\. Wang, Y\. Wang, and L\. WangCoDiff: conditional diffusion model for collaborative 3d object detection\.External Links:2502\.14891,[Link](https://arxiv.org/abs/2502.14891)Cited by:[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p1.1)\.
- Huang and Gool \(2017\)Z\. Huang and L\. V\. GoolA riemannian network for spd matrix learning\.InProceedings of the 31st AAAI Conference on Artificial Intelligence \(AAAI\),External Links:1608\.04233,[Link](https://arxiv.org/abs/1608.04233)Cited by:[§3\.2](https://arxiv.org/html/2609.03818#S3.SS2.SSS0.Px2.p1.1)\.
- Langet al\.\(2019\)A\. H\. Lang, S\. Vora, H\. Caesar, L\. Zhou, J\. Yang, and O\. BeijbomPointPillars: fast encoders for object detection from point clouds\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:1812\.05784,[Link](https://arxiv.org/abs/1812.05784)Cited by:[§4\.1](https://arxiv.org/html/2609.03818#S4.SS1.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)M\. Li, H\. Lin, L\. Qiu, X\. Liang, L\. Chen, A\. Elsaddik, and X\. ChangContrastive learning with counterfactual explanations for radiology report generation\.InProceedings of the European Conference on Computer Vision \(ECCV\),External Links:2407\.14474,[Link](https://arxiv.org/abs/2407.14474)Cited by:[§2\.2](https://arxiv.org/html/2609.03818#S2.SS2.p1.1)\.
- Liet al\.\(2017\)Y\. Li, C\. Fang, J\. Yang, Z\. Wang, X\. Lu, and M\. YangUniversal style transfer via feature transforms\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:1705\.08086,[Link](https://arxiv.org/abs/1705.08086)Cited by:[§3\.2](https://arxiv.org/html/2609.03818#S3.SS2.SSS0.Px2.p1.1)\.
- Liet al\.\(2021\)Y\. Li, S\. Ren, P\. Wu, S\. Chen, C\. Feng, and W\. ZhangLearning distilled collaboration graph for multi\-agent perception\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.34\.External Links:2111\.00643,[Link](https://arxiv.org/abs/2111.00643)Cited by:[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p1.1)\.
- Liuet al\.\(2023\)S\. Liu, C\. Gao, Y\. Chen, X\. Peng, X\. Kong, K\. Wang, R\. Xu, W\. Jiang, H\. Xiang, J\. Ma, and M\. WangTowards vehicle\-to\-everything autonomous driving: a survey on collaborative perception\.External Links:2308\.16714,[Link](https://arxiv.org/abs/2308.16714)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p1.1)\.
- Liuet al\.\(2020\)Y\. Liu, J\. Tian, N\. Glaser, and Z\. KiraWhen2com: multi\-agent perception via communication graph grouping\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:2006\.00176,[Link](https://arxiv.org/abs/2006.00176)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p1.1)\.
- Liuet al\.\(2022\)Z\. Liu, H\. Mao, C\. Wu, C\. Feichtenhofer, T\. Darrell, and S\. XieA convnet for the 2020s\.Note:Published in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\), 2022External Links:2201\.03545,[Link](https://arxiv.org/abs/2201.03545)Cited by:[§B\.2](https://arxiv.org/html/2609.03818#A2.SS2.p1.1),[§3\.3](https://arxiv.org/html/2609.03818#S3.SS3.p2.2)\.
- Luet al\.\(2024\)Y\. Lu, Y\. Hu, Y\. Zhong, D\. Wang, S\. Chen, and Y\. WangAn extensible framework for open heterogeneous collaborative perception\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p2.1),[§4\.2](https://arxiv.org/html/2609.03818#S4.SS2.SSS0.Px1.p1.1)\.
- Luet al\.\(2023\)Y\. Lu, Q\. Li, B\. Liu, M\. Dianati, C\. Feng, S\. Chen, and Y\. WangRobust collaborative 3d object detection in presence of pose errors\.InProceedings of the IEEE International Conference on Robotics and Automation \(ICRA\),External Links:2211\.07214,[Link](https://arxiv.org/abs/2211.07214)Cited by:[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p1.1)\.
- Luoet al\.\(2024\)T\. Luo, Q\. Yuan, G\. Luo, Y\. Xia, Y\. Yang, and J\. LiPlug and Play: a representation enhanced domain adapter for collaborative perception\.InComputer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXXXI,Berlin, Heidelberg,pp\. 287–303\.External Links:ISBN 978\-3\-031\-73003\-0,[Link](https://doi.org/10.1007/978-3-031-73004-7_17),[Document](https://dx.doi.org/10.1007/978-3-031-73004-7%5F17)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p2.1),[§4\.2](https://arxiv.org/html/2609.03818#S4.SS2.SSS0.Px1.p1.1)\.
- Philion and Fidler \(2020\)J\. Philion and S\. FidlerLift, splat, shoot: encoding images from arbitrary camera rigs by implicitly unprojecting to 3d\.InProceedings of the European Conference on Computer Vision \(ECCV\),External Links:2008\.05711,[Link](https://arxiv.org/abs/2008.05711)Cited by:[§4\.1](https://arxiv.org/html/2609.03818#S4.SS1.SSS0.Px2.p1.1)\.
- Qiaoet al\.\(2025\)Z\. Qiao, C\. Liu, Y\. Zhang, M\. Jin, Q\. Pham, Q\. Wen, P\. N\. Suganthan, X\. Jiang, and S\. RamasamyMulti\-scale finetuning for encoder\-based time series foundation models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2506\.14087,[Link](https://arxiv.org/abs/2506.14087)Cited by:[§2\.2](https://arxiv.org/html/2609.03818#S2.SS2.p1.1)\.
- Schölkopf \(2022\)B\. SchölkopfCausality for machine learning\.InProbabilistic and Causal Inference,pp\. 765–804\.External Links:ISBN 9781450395861,[Link](http://dx.doi.org/10.1145/3501714.3501755),[Document](https://dx.doi.org/10.1145/3501714.3501755)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p6.1)\.
- Shaoet al\.\(2024\)C\. Shao, G\. Luo, Q\. Yuan, Y\. Chen, Y\. Liu, K\. Gong, and J\. LiHetecooper: feature collaboration graph for heterogeneous collaborative perception\.InEuropean Conference on Computer Vision,External Links:[Link](https://api.semanticscholar.org/CorpusID:272350894)Cited by:[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p2.1)\.
- Shaoet al\.\(2025\)C\. Shao, Q\. Yuan, G\. Luo, Y\. Hu, D\. Wang, Y\. Liu, R\. Pan, B\. Chen, and J\. LiNegoCollab: a common representation negotiation approach for heterogeneous collaborative perception\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2510\.27647,[Link](https://arxiv.org/abs/2510.27647)Cited by:[§C\.3](https://arxiv.org/html/2609.03818#A3.SS3.p1.1),[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p2.1)\.
- Tan and Le \(2019\)M\. Tan and Q\. V\. LeEfficientNet: rethinking model scaling for convolutional neural networks\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),External Links:1905\.11946,[Link](https://arxiv.org/abs/1905.11946)Cited by:[§4\.1](https://arxiv.org/html/2609.03818#S4.SS1.SSS0.Px2.p1.1)\.
- van den Oordet al\.\(2019\)A\. van den Oord, Y\. Li, and O\. VinyalsRepresentation learning with contrastive predictive coding\.External Links:1807\.03748,[Link](https://arxiv.org/abs/1807.03748)Cited by:[§3\.2](https://arxiv.org/html/2609.03818#S3.SS2.SSS0.Px5.p3.1)\.
- Wanget al\.\(2024\)L\. Wang, Z\. He, R\. Dang, M\. Shen, C\. Liu, and Q\. ChenVision\-and\-language navigation via causal learning\.External Links:2404\.10241,[Link](https://arxiv.org/abs/2404.10241)Cited by:[§2\.2](https://arxiv.org/html/2609.03818#S2.SS2.p1.1)\.
- Weiet al\.\(2023\)S\. Wei, Y\. Wei, Y\. Hu, Y\. Lu, Y\. Zhong, S\. Chen, and Y\. ZhangAsynchrony\-robust collaborative perception via bird’s eye view flow\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2309\.16940,[Link](https://arxiv.org/abs/2309.16940)Cited by:[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p1.1)\.
- Xianget al\.\(2023\)H\. Xiang, R\. Xu, and J\. MaHM\-vit: hetero\-modal vehicle\-to\-vehicle cooperative perception with vision transformer\.External Links:2304\.10628,[Link](https://arxiv.org/abs/2304.10628)Cited by:[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p2.1)\.
- Xuet al\.\(2021\)R\. Xu, Y\. Guo, X\. Han, X\. Xia, H\. Xiang, and J\. MaOpenCDA:an open cooperative driving automation framework integrated with co\-simulation\.External Links:2107\.06260,[Link](https://arxiv.org/abs/2107.06260)Cited by:[§A\.1](https://arxiv.org/html/2609.03818#A1.SS1.SSS0.Px1.p1.1)\.
- Xuet al\.\(2023\)R\. Xu, J\. Li, X\. Dong, H\. Yu, and J\. MaBridging the domain gap for multi\-agent perception\.InProceedings of the IEEE International Conference on Robotics and Automation \(ICRA\),External Links:2210\.08451,[Link](https://arxiv.org/abs/2210.08451)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p2.1),[§4\.2](https://arxiv.org/html/2609.03818#S4.SS2.SSS0.Px1.p1.1)\.
- Xuet al\.\(2022a\)R\. Xu, Z\. Tu, H\. Xiang, W\. Shao, B\. Zhou, and J\. MaCoBEVT: cooperative bird’s eye view semantic segmentation with sparse transformers\.InProceedings of the 6th Conference on Robot Learning \(CoRL\),External Links:2207\.02202,[Link](https://arxiv.org/abs/2207.02202)Cited by:[§2\.1](https://arxiv.org/html/2609.03818#S2.SS1.p1.1)\.
- Xuet al\.\(2022b\)R\. Xu, H\. Xiang, Z\. Tu, X\. Xia, M\. Yang, and J\. MaV2X\-vit: vehicle\-to\-everything cooperative perception with vision transformer\.InComputer Vision – ECCV 2022,S\. Avidan, G\. Brostow, M\. Cissé, G\. M\. Farinella, and T\. Hassner \(Eds\.\),Lecture Notes in Computer Science, Vol\.13699,pp\. 107–124\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-19842-7%5F7)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p1.1)\.
- Xuet al\.\(2022c\)R\. Xu, H\. Xiang, X\. Xia, X\. Han, J\. Li, and J\. MaOPV2V: an open benchmark dataset and fusion pipeline for perception with vehicle\-to\-vehicle communication\.InProceedings of the IEEE International Conference on Robotics and Automation \(ICRA\),External Links:2109\.07644,[Link](https://arxiv.org/abs/2109.07644)Cited by:[§A\.1](https://arxiv.org/html/2609.03818#A1.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.03818#S4.SS1.SSS0.Px1.p1.1)\.
- Yanet al\.\(2024\)J\. Yan, C\. Deng, H\. Huang, and W\. LiuCausality\-invariant interactive mining for cross\-modal similarity learning\.IEEE Transactions on Pattern Analysis and Machine Intelligence \(TPAMI\)46\(9\),pp\. 6216–6230\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2024.3379752),[Link](https://doi.org/10.1109/TPAMI.2024.3379752)Cited by:[§2\.2](https://arxiv.org/html/2609.03818#S2.SS2.p1.1)\.
- Yanet al\.\(2018\)Y\. Yan, Y\. Mao, and B\. LiSecond: sparsely embedded convolutional detection\.Sensors18\(10\),pp\. 3337\.Cited by:[§4\.1](https://arxiv.org/html/2609.03818#S4.SS1.SSS0.Px2.p1.1)\.
- Yinet al\.\(2024\)H\. Yin, D\. Tian, C\. Lin, X\. Duan, J\. Zhou, D\. Zhao, and D\. CaoV2VFormer\+\+: multi\-modal vehicle\-to\-vehicle cooperative perception via global\-local transformer\.IEEE Transactions on Intelligent Transportation Systems25\(2\),pp\. 2153–2166\.External Links:[Document](https://dx.doi.org/10.1109/TITS.2023.3314919)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p2.1)\.
- Yuet al\.\(2022\)H\. Yu, Y\. Luo, M\. Shu, Y\. Huo, Z\. Yang, Y\. Shi, Z\. Guo, H\. Li, X\. Hu, J\. Yuan, and Z\. NieDAIR\-v2x: a large\-scale dataset for vehicle\-infrastructure cooperative 3d object detection\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:2204\.05575,[Link](https://arxiv.org/abs/2204.05575)Cited by:[§A\.1](https://arxiv.org/html/2609.03818#A1.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.03818#S4.SS1.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)T\. Zhang, P\. Liu, Y\. Lu, M\. Cai, Z\. Zhang, Z\. Zhang, and Q\. ZhouCWNet: causal wavelet network for low\-light image enhancement\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),External Links:2507\.10689,[Link](https://arxiv.org/abs/2507.10689)Cited by:[§2\.2](https://arxiv.org/html/2609.03818#S2.SS2.p1.1)\.
- Zhaoet al\.\(2023\)B\. Zhao, W\. Zhang, and Z\. ZouBM2CP: efficient collaborative perception with lidar\-camera modalities\.InProceedings of the 7th Conference on Robot Learning \(CoRL\),External Links:2310\.14702,[Link](https://arxiv.org/abs/2310.14702)Cited by:[§1](https://arxiv.org/html/2609.03818#S1.p2.1)\.
- Zhou and Tuzel \(2017\)Y\. Zhou and O\. TuzelVoxelNet: end\-to\-end learning for point cloud based 3d object detection\.External Links:1711\.06396,[Link](https://arxiv.org/abs/1711.06396)Cited by:[§A\.3](https://arxiv.org/html/2609.03818#A1.SS3.p1.1),[§C\.4](https://arxiv.org/html/2609.03818#A3.SS4.p1.1)\.

## Appendix ADetails of Experiment

### A\.1Dataset

##### OPV2V\.

OPV2V\([Xu et al\., 2022c](https://arxiv.org/html/2609.03818#bib.bib31)\)is a large\-scale public dataset for multi\-agent collaborative perception research\. Generated by the CARLA\([Dosovitskiy et al\., 2017](https://arxiv.org/html/2609.03818#bib.bib38)\)simulation platform combined with the OpenCDA\([Xu et al\., 2021](https://arxiv.org/html/2609.03818#bib.bib39)\)co\-simulation framework, it contains over 70 scenarios with a total of 11,464 frames, covering diverse urban scenarios and various road structures\. Each agent is equipped with one 16\-channel, one 32\-channel, and one 64\-channel LiDAR, along with 4 RGB cameras and 4 depth cameras\.

##### DAIR\-V2X\.

DAIR\-V2X\([Yu et al\., 2022](https://arxiv.org/html/2609.03818#bib.bib32)\)is a real\-world dataset dedicated to vehicle\-infrastructure collaborative perception\. The dataset scenarios consist of one vehicle and one Road Side Unit \(RSU\)\. The agent is equipped with a 40\-channel LiDAR and a 1920×1080 resolution camera, while the RSU is equipped with a 300\-channel LiDAR and a camera of the same specification\.

### A\.2Training Details

To ensure fair comparisons, we use identical encoder weights across different methods, such that the modality encoders achieve the same detection performance in homogeneous settings\. All LiDAR encoders are trained with the perception range of\[−102\.4,−51\.2,102\.4,51\.2\]\[\-102\.4,\-51\.2,102\.4,51\.2\], while all camera encoders are trained with\[−51\.2,−51\.2,51\.2,51\.2\]\[\-51\.2,\-51\.2,51\.2,51\.2\]\. All heterogeneous cooperative perception networks are trained and evaluated within\[−51\.2,−51\.2,51\.2,51\.2\]\[\-51\.2,\-51\.2,51\.2,51\.2\]\. Our training is conducted on NVIDIA RTX 4090 GPUs\. We use the Adam optimizer with an initial learning rate of1×10−31\\times 10^\{\-3\}\. The unified converter is trained for 10 epochs\. Adapting to a new modality only requires 6 epochs to reach aligned semantics\. Each modality\-specific local semantic reconstructor is trained for 5 epochs\. The hyperparameters used during training are shown in Table[5](https://arxiv.org/html/2609.03818#A1.T5)\.

Table 5:Hyperparameters in experiments\.Stage\-1Stage\-2Loss Weightλcon\\lambda\_\{\\mathrm\{con\}\}λctx\\lambda\_\{\\mathrm\{ctx\}\}λssim\\lambda\_\{\\mathrm\{ssim\}\}λtask\(1\)\\lambda\_\{\\mathrm\{task\}\}^\{\(1\)\}λtask\(2\)\\lambda\_\{\\mathrm\{task\}\}^\{\(2\)\}λmse\\lambda\_\{\\mathrm\{mse\}\}λcons\\lambda\_\{\\mathrm\{cons\}\}Value2\.00\.51\.00\.51\.05\.05\.0
### A\.3Detailed Configuration of Agents

Our experiments involve eight agent modalities and a protocol modality, whose perception configurations and performance in homogeneous scenarios are shown in Table[6](https://arxiv.org/html/2609.03818#A1.T6)\. Here,LLdenotes the LiDAR modality, andCCdenotes the Camera modality\. In the supplementary experiments,Vv​n​4V\_\{vn4\}\([Zhou and Tuzel, 2017](https://arxiv.org/html/2609.03818#bib.bib2)\)andCRes34C\_\{\\mathrm\{Res34\}\}\([He et al\., 2015](https://arxiv.org/html/2609.03818#bib.bib29)\)are additionally included\.

Table 6:Configuration and performance of agentsAgentEncoderVoxel SizeCamera\-EncoderAP@0\.3 / AP@0\.5↑\\uparrowProtocolPointPillar0\.4, 0\.4, 4\-0\.982 / 0\.980Lp​p​4L\_\{pp4\}PointPillar0\.4, 0\.4, 4\-0\.972 / 0\.968CEffC\_\{\\mathrm\{Eff\}\}Lift\-Splat\-Shoot\-EfficientNet0\.790 / 0\.706Ls​d​1L\_\{sd1\}SECOND0\.1, 0\.1, 0\.1\-0\.975 / 0\.971CResC\_\{\\mathrm\{Res\}\}Lift\-Splat\-Shoot\-ResNet101\(Output Channel = 512\)0\.729 / 0\.625Ls​d​2L\_\{sd2\}SECOND0\.2, 0\.2, 0\.2\-0\.973 / 0\.968CEffB1C\_\{\\mathrm\{EffB1\}\}Lift\-Splat\-Shoot\-EfficientNetB10\.652 / 0\.579Lv​n​4L\_\{vn4\}VoxelNet0\.4, 0\.4, 0\.4\-0\.960 / 0\.958CRes34C\_\{\\mathrm\{Res34\}\}Lift\-Splat\-Shoot\-ResNet34\(Output Channel = 128\)0\.619 / 0\.546

## Appendix BImplementation Details

### B\.1Intervention via SPD Feature Geometry

All operations are performed on the reshaped protocol feature

p∈ℝC×H×W→X∈ℝC×N,N=H⋅W,p\\in\\mathbb\{R\}^\{C\\times H\\times W\}\\ \\rightarrow\\ X\\in\\mathbb\{R\}^\{C\\times N\},\\quad N=H\\cdot W,where thenn\-th columnxn∈ℝCx\_\{n\}\\in\\mathbb\{R\}^\{C\}denotes theCC\-channel feature vector at spatial positionnn\.

We represent modality style using the weighted first\- and second\-order statistics\(μ,Σ\)\(\\mu,\\Sigma\)computed fromXX, withΣ∈𝒮\+C\\Sigma\\in\\mathcal\{S\}\_\{\+\}^\{C\}\. A semantic maskm∈\[0,1\]1×Nm\\in\[0,1\]^\{1\\times N\}, produced by a mask generatorg⁡\(⋅\)g\(\\cdot\), is*required*to define spatially varying non\-negative weights, thereby modulating foreground and background contributions\. The Mask Generator applies the classification head from single\-modality training to local features to produce a confidence map, which is filtered by a Gaussian kernel \(k=5,σ=1\.0k=5,\\sigma=1\.0\), and thresholded at 0\.01 to obtain a binary mask\.

##### Notation and weighted statistics\.

GivenX=\[x1,…,xN\]X=\[x\_\{1\},\\dots,x\_\{N\}\]and non\-negative weightsw=\[w1,…,wN\]w=\[w\_\{1\},\\dots,w\_\{N\}\], we compute

μ\\displaystyle\\mu=∑n=1Nwn​xn∑n=1Nwn,\\displaystyle=\\frac\{\\sum\_\{n=1\}^\{N\}w\_\{n\}x\_\{n\}\}\{\\sum\_\{n=1\}^\{N\}w\_\{n\}\},\(15\)Σ\\displaystyle\\Sigma=∑n=1Nwn​\(xn−μ\)​\(xn−μ\)⊤∑n=1Nwn\+ε​I,ε\>0,\\displaystyle=\\frac\{\\sum\_\{n=1\}^\{N\}w\_\{n\}\(x\_\{n\}\-\\mu\)\(x\_\{n\}\-\\mu\)^\{\\top\}\}\{\\sum\_\{n=1\}^\{N\}w\_\{n\}\}\+\\varepsilon I,\\qquad\\varepsilon\>0,\(16\)whereε​I\\varepsilon Iensures numerical stability\. In practice, the weights are parameterized using the semantic mask as

w=η\+ρ​m,η\>0,ρ≥0,w=\\eta\+\\rho\\,m,\\qquad\\eta\>0,\\ \\rho\\geq 0,\(17\)which guarantees strictly positive weights and allows the mask to control contribution strength\.

##### SPD and whitening–coloring transform\.

For any SPD matrixΣ\\Sigma, letΣ=U​Λ​U⊤\\Sigma=U\\Lambda U^\{\\top\}be its eigendecomposition and define

Σ12=U​Λ12​U⊤,Σ−12=U​\(Λ\+ε​I\)−12​U⊤,\\Sigma^\{\\tfrac\{1\}\{2\}\}=U\\Lambda^\{\\tfrac\{1\}\{2\}\}U^\{\\top\},\\qquad\\Sigma^\{\-\\tfrac\{1\}\{2\}\}=U\(\\Lambda\+\\varepsilon I\)^\{\-\\tfrac\{1\}\{2\}\}U^\{\\top\},whereΛ12\\Lambda^\{\\tfrac\{1\}\{2\}\}is applied elementwise andε\\varepsilonis added to the eigenvalues for stability\. Given source statistics\(μs,Σs\)\(\\mu\_\{s\},\\Sigma\_\{s\}\)and target statistics\(μt,Σt\)\(\\mu\_\{t\},\\Sigma\_\{t\}\), the closed\-form whitening–coloring transform applied toXXis

Ts→t​\(X\)=μt\+Σt12​Σs−12​\(X−μs\)\.T\_\{s\\to t\}\(X\)\\;=\\;\\mu\_\{t\}\+\\Sigma\_\{t\}^\{\\tfrac\{1\}\{2\}\}\\,\\Sigma\_\{s\}^\{\-\\tfrac\{1\}\{2\}\}\\,\(X\-\\mu\_\{s\}\)\.\(18\)This linear mapping preserves the spatial arrangement of columns inXXwhile modifying only channel statistics\.

#### Mask\-aware Canonical Suppression \(MCS\)

The objective of MCS is to demodalize a protocol feature by mapping its channel statistics to a neutral canonical distribution\(μ0,Σ0\)\(\\mu\_\{0\},\\Sigma\_\{0\}\), while preserving spatial semantics through a mask\-guided residual update\.

Algorithm 1Mask\-aware Canonical Suppression \(MCS\)0:

X∈ℝC×NX\\in\\mathbb\{R\}^\{C\\times N\}\(protocol feature\), confidence map

MM
0:

X0X\_\{0\}\(modality\-agnostic feature\)

1:

m←g⁡\(M\)m\\leftarrow g\(M\)\{semantic mask,

1×N1\\times N\}

2:

w←η\+ρ​mw\\leftarrow\\eta\+\\rho\\,m\{weights as in \([17](https://arxiv.org/html/2609.03818#A2.E17)\)\}

3:Compute

\(μc,Σc\)\(\\mu\_\{c\},\\Sigma\_\{c\}\)from

XXusing \([15](https://arxiv.org/html/2609.03818#A2.E15)\)–\([16](https://arxiv.org/html/2609.03818#A2.E16)\)

4:Set the canonical target to

\(μ0,Σ0\)=\(𝟎,I\)\(\\mu\_\{0\},\\Sigma\_\{0\}\)=\(\\mathbf\{0\},I\)
5:

X~←Tc→0​\(X\)\\tilde\{X\}\\leftarrow T\_\{c\\to 0\}\(X\)using \([18](https://arxiv.org/html/2609.03818#A2.E18)\)

6:Compute strength map

β⁡\(m\)∈\[0,1\]1×N\\beta\(m\)\\in\[0,1\]^\{1\\times N\}
7:

X0←X\+\(β⁡\(m\)⊙𝟏C\)⊙\(X~−X\)X\_\{0\}\\leftarrow X\+\(\\beta\(m\)\\odot\\mathbf\{1\}\_\{C\}\)\\odot\(\\tilde\{X\}\-X\)
8:Return:

X0X\_\{0\}

Note: All eigendecompositions apply a small eigenvalue regularizer to ensureΣ−12\\Sigma^\{\-\{\\tfrac\{1\}\{2\}\}\}is well\-defined\.

#### Canonical Suppress\-Then\-Inject \(STI\)

To create hard negative samples and controlled modality perturbations, we first suppress modality bias via MCS and then inject heterogeneous modality statistics\(μs,Σs\)\(\\mu\_\{s\},\\Sigma\_\{s\}\)computed from a style featureXsX\_\{s\}\. The same semantic maskmm\(and associated weightsww\) is used so that injection respects the original spatial semantics\.

Algorithm 2Canonical Suppress\-Then\-Inject \(STI\)0:

XX\(content protocol feature\),

XsX\_\{s\}\(heterogeneous style feature\), confidence map

MM
0:

X\+X^\{\+\}\(modality\-agnostic positive\),

X−X^\{\-\}\(injected hard negative\)

1:

X0←MCS​\(X,M\)X\_\{0\}\\leftarrow\\text\{MCS\}\(X,M\)
2:

m←g⁡\(M\),w←η\+ρ​mm\\leftarrow g\(M\),\\;w\\leftarrow\\eta\+\\rho m
3:Compute

\(μ0,Σ0\)\(\\mu\_\{0\},\\Sigma\_\{0\}\)from

X0X\_\{0\}and

\(μs,Σs\)\(\\mu\_\{s\},\\Sigma\_\{s\}\)from

XsX\_\{s\}
4:

X^←T0→s​\(X0\)\\hat\{X\}\\leftarrow T\_\{0\\to s\}\(X\_\{0\}\)
5:Compute injection strength map

βinj​\(m\)\\beta^\{\\text\{inj\}\}\(m\)
6:

X−←X0\+\(βinj​\(m\)⊙𝟏C\)⊙\(X^−X0\)X^\{\-\}\\leftarrow X\_\{0\}\+\(\\beta^\{\\text\{inj\}\}\(m\)\\odot\\mathbf\{1\}\_\{C\}\)\\odot\(\\hat\{X\}\-X\_\{0\}\)
7:

X\+←X0X^\{\+\}\\leftarrow X\_\{0\}
8:Return:

X\+,X−X^\{\+\},X^\{\-\}

The positive samplep\+p^\{\+\}is obtained by reshapingX\+X^\{\+\}back to the original spatial layout and serves as the modality\-agnostic anchor for metric learning\. The negative samplep−p^\{\-\}is the injected featureX−X^\{\-\}, which is a hard negative created by re\-introducing controlled heterogeneous modality statistics while preserving consistent semantics\.

### B\.2SCE Architecture

The architecture of the SCE is illustrated in Figure[5](https://arxiv.org/html/2609.03818#A2.F5), which gradually enhances the global semantic modeling capability through two\-step downsampling and ConvNeXt\([Liu et al\., 2022](https://arxiv.org/html/2609.03818#bib.bib5)\)\.

Figure 5:Detailed structure of the Semantic Context Extractor
### B\.3CGDR Architecture

The architecture of the CGDR is illustrated in Figure[6](https://arxiv.org/html/2609.03818#A2.F6)\. This module takes the local modality featurefi∈ℝC×H×Wf\_\{i\}\\in\\mathbb\{R\}^\{C\\times H\\times W\}and the context featurec^i∈ℝCc​t​x×H×W\\hat\{c\}\_\{i\}\\in\\mathbb\{R\}^\{C\_\{ctx\}\\times H\\times W\}as inputs\. After the context feature is processed by a1×11\\times 1*Project Conv*, it is concatenated withfif\_\{i\}and fed into the*Joint Fusion*module, which consists of a depth\-wise convolution and LayerNorm, for information integration\.

The fused feature is modulated by the*Context Guide*module \(FiLM\-style multi\-head feature modulation\) and recalibrated in channel dimension via the*SE*module\([Hu et al\., 2019](https://arxiv.org/html/2609.03818#bib.bib4)\), then filtered by the*Gate*module \(1x1 convolution followed by SiLU\) to suppress noise\. Subsequently, a residual connection with Layer Scale stabilizes the gradient, followed by transformation through an*MLP*and another residual connection with Layer Scale 2 to preserve detailed features\. Finally, the output is split into an updated local feature and a context feature for subsequent layers, which balances local discriminability and global semantic consistency\.

Figure 6:Detailed structure of the Context\-Guided Dynamic Refiner

## Appendix CMore Experiments

### C\.1Comparison between CauseCollab and Late Fusion

Late fusion directly integrates the 3D detection boxes perceived by each agent to enable information sharing\.We compare the performance of our method with that of late fusion on the OPV2V dataset under different pose errorσ\\sigmavalues\. As shown in Table[7](https://arxiv.org/html/2609.03818#A3.T7),*ALL*denotes the collaboration of the four modalities\. We additionally set two modality configurations for Ego\-Neighbor pairs with large modality gaps\. Our method significantly outperforms the late fusion method across all pose error levels and collaboration combinations, demonstrating the superiority of our method in real\-world application scenarios\.

Table 7:Performance comparison of CauseCollab and Late Fusion under varying pose error standard deviationsσ\\sigmaMethodAP@30AP@50ALLLp​p​4\+CEffL\_\{pp4\}\+C\_\{\\mathrm\{Eff\}\}CEff\+CResC\_\{\\mathrm\{Eff\}\}\+C\_\{\\mathrm\{Res\}\}ALLLp​p​4\+CEffL\_\{pp4\}\+C\_\{\\mathrm\{Eff\}\}CEff\+CResC\_\{\\mathrm\{Eff\}\}\+C\_\{\\mathrm\{Res\}\}0\.0Late Fusion0\.9570\.9560\.7450\.9430\.9300\.630CauseCollab0\.9780\.9710\.7720\.9750\.9660\.6700\.2Late Fusion0\.9520\.9510\.7340\.8480\.8510\.534CauseCollab0\.9760\.9700\.7620\.9680\.9600\.6250\.4Late Fusion0\.8300\.8490\.6240\.5630\.6320\.332CauseCollab0\.9640\.9600\.6980\.9160\.9220\.4870\.6Late Fusion0\.6790\.7300\.4840\.4370\.5380\.235CauseCollab0\.9280\.9320\.5900\.8590\.8820\.381
### C\.2Analysis of Different Protocol Modalities

The alignment effectiveness of protocol\-based heterogeneous collaborative perception depends on the affinity between the protocol modality and heterogeneous modalities\. To analyze this effect, we conduct supplementary experiments using either LiDAR \(Lp​p​4L\_\{pp4\}\) or Camera \(CE​f​fC\_\{Eff\}\) as the protocol modality\. The results are shown in Table[8](https://arxiv.org/html/2609.03818#A3.T8)\.

Table 8:Performance comparison under different protocol modalities\.Protocol ModalityALLLpp4\+CEffL\_\{\\mathrm\{pp4\}\}\+C\_\{\\mathrm\{Eff\}\}CEff\+Lpp4C\_\{\\mathrm\{Eff\}\}\+L\_\{\\mathrm\{pp4\}\}LiDAR \(PointPillar\)0\.978 / 0\.9750\.971 / 0\.9660\.867 / 0\.842Camera \(Lift\-Splat\-Shoot\)0\.975 / 0\.9730\.968 / 0\.9640\.875 / 0\.847The results show that CauseCollab achieves comparable collaborative perception performance under both LiDAR\-based and Camera\-based protocol modalities\. This indicates that the proposed causal intervention effectively reduces protocol\-modality asymmetry and improves the robustness of the learned protocol space\.

### C\.3Comparison between CauseCollab and NegoCollab

NegoCollab\([Shao et al\., 2025](https://arxiv.org/html/2609.03818#bib.bib20)\)mitigates the domain gap by negotiating a common representation, whereas our method explicitly disentangles semantic factors from modal entanglement to learn a modality\-agnostic protocol space\. Experimental results demonstrate that our method outperforms NegoCollab in both general and novel\-modality scenarios\.ALLrepresents the general multi\-agent collaboration scenarios under the same experimental settings as in Table[1](https://arxiv.org/html/2609.03818#S3.T1), whileLs​d​2new\+CEffB1newL\_\{sd2^\{\\text\{new\}\}\}\+C\_\{\\mathrm\{EffB1\}^\{\\text\{new\}\}\}verifies the performance on the new modality\.

Table 9:Performance and parameter comparison of CauseCollab and NegoCollabMethodALLLs​d​2new\+CEffB1newL\_\{sd2^\{\\text\{new\}\}\}\+C\_\{\\mathrm\{EffB1\}^\{\\text\{new\}\}\}\#Params \(MB\)CauseCollab0\.978/0\.9750\.966/0\.9590\.68NegoCollab0\.974/0\.9720\.958/0\.9530\.98
### C\.4Robustness Evaluation on More Novel Modality Adaptation

To further validate the robustness of the CauseCollab protocol space for novel modality adaptation, we introduce two additional novel modalities:Lvn4newL\_\{\\mathrm\{vn4\}^\{\\text\{new\}\}\}andCRes34newC\_\{\\mathrm\{Res34\}\}^\{\\mathrm\{new\}\}\. Specifically,Lvn4newL\_\{\\mathrm\{vn4\}^\{\\text\{new\}\}\}denotes a LiDAR modality encoded by VoxelNet\([Zhou and Tuzel, 2017](https://arxiv.org/html/2609.03818#bib.bib2)\), whileCRes34newC\_\{\\mathrm\{Res34\}^\{\\text\{new\}\}\}denotes a Camera modality based on ResNet34 with a reduced output channel size of 128\. Together withLsd2newL\_\{\\mathrm\{sd2\}^\{\\text\{new\}\}\}andCEffB1newC\_\{\\mathrm\{EffB1\}^\{\\text\{new\}\}\}, we evaluate a collaborative perception scenario involving four novel modalities:Lvn4new\+CRes34new\+Lsd2new\+CEffB1newL\_\{\\mathrm\{vn4\}^\{\\text\{new\}\}\}\+C\_\{\\mathrm\{Res34\}^\{\\text\{new\}\}\}\+L\_\{\\mathrm\{sd2\}^\{\\text\{new\}\}\}\+C\_\{\\mathrm\{EffB1\}^\{\\text\{new\}\}\}\. The results are reported in Table[10](https://arxiv.org/html/2609.03818#A3.T10)\.

Table 10:Robustness evaluation under four novel modalities\.MethodAP@30AP@50MPDA0\.9700\.968PnPDA0\.9680\.965HEAL0\.9190\.917STAMP0\.9630\.961CauseCollab0\.9740\.971
### C\.5Additional qualitative experiments

Figure[7](https://arxiv.org/html/2609.03818#A3.F7)presents additional visualization experiments\. To highlight the visual differences, we specifically selectCEffC\_\{\\mathrm\{Eff\}\}andCResC\_\{\\mathrm\{Res\}\}for collaboration due to their larger modality gaps\. The visualization intuitively demonstrates the robustness of our method even in such scenarios with larger domain gaps\.

Scenario 1Scenario 2Scenario 3Scenario 4MPDA![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/mpda/bev_01960.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/mpda/bev_00520.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/mpda/bev_00600.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/mpda/bev_02000.png)PnPDA![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/pnpda/bev_01960.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/pnpda/bev_00520.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/pnpda/bev_00600.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/pnpda/bev_02000.png)HEAL![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/heal/bev_01960.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/heal/bev_00520.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/heal/bev_00600.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/heal/bev_02000.png)STAMP![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/stamp/bev_01960.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/stamp/bev_00520.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/stamp/bev_00600.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/stamp/bev_02000.png)CauseCollab![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/cause/bev_01960.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/cause/bev_00520.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/cause/bev_00600.png)![Refer to caption](https://arxiv.org/html/2609.03818v1/figs/add_vis/cause/bev_02000.png)Figure 7:Visual comparison of detection results across different methods, usingCEffC\_\{\\mathrm\{Eff\}\}andCResC\_\{\\mathrm\{Res\}\}\.

Similar Articles

Generalized Multimodal Foundation Model

arXiv cs.LG

This paper introduces a generalized multimodal foundation model capable of handling arbitrary modality combinations and prediction tasks, achieving competitive performance through training on large-scale synthetic datasets with diverse causal structures.

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

Hugging Face Daily Papers

CausaLab is a scalable environment for evaluating LLM agents on interactive causal discovery, assessing both predictive accuracy and faithful recovery of underlying causal mechanisms. Experiments reveal a gap between prediction and mechanism recovery, highlighting limits in current LLM agents as experimental causal reasoners.