A Hybrid Mamba for Audio-Visual Navigation

# A Hybrid Mamba for Audio-Visual Navigation†
Source: [https://arxiv.org/html/2607.13110](https://arxiv.org/html/2607.13110)
Yi Wang1,2,3, and Yinfeng Yu1,2,3,🖂\{\}^\{,\\mbox\{\\Letter\}\}†\\daggerThis research was financially supported by the National Natural Science Foundation of China \(Grant No\. 62463029\)\.1School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China\.2Joint Research Laboratory for Embodied Intelligence, Xinjiang University\.3Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University\.🖂\{\}^\{\\mbox\{\\Letter\}\}Yinfeng Yu is the corresponding author \(Email: yuyinfeng@xju\.edu\.cn\)\.

###### Abstract

Since the paradigm centered on convolutional neural networks and recurrent architectures was established in 2020, the fundamental backbone networks for audio\-visual navigation have undergone no essential changes for more than five years, making them inadequate to support efficient representation of dynamic multimodal sequences\. This paper proposes Samba\(A Hybrid Mamba for Audio\-Visual Navigation\)\. It uses the adaptive selection\-enabled Mamba State Encoder \(M\-SE\) to replace conventional GRUs for temporal aggregation, and constructs an Audio Mamba Encoder \(AME\) to remedy the limitations of convolutional operators in capturing global time\-frequency dependencies in spectrograms\. Experiments demonstrate that Samba exhibits exceptional generalization performance when facing unheard sound sources and unseen scenes\. On the Matterport3D dataset, it improves the navigation success rate \(SR\) by 11\.3% compared with existing state\-of\-the\-art models, and the performance gain is even more pronounced on the Replica dataset, which features finer scene structures\. Such modernized architectural reconstruction unlocks stronger embodied representation capabilities at a lower computational cost, thereby providing a highly robust technical pathway for paradigm evolution in the field of audio\-visual navigation\.

## IINTRODUCTION

Audio\-visual navigation \(AVN\)\[[5](https://arxiv.org/html/2607.13110#bib.bib1),[9](https://arxiv.org/html/2607.13110#bib.bib2)\]requires agents to localize sound sources in unmapped environments by coupling local visual geometry with long\-range auditory guidance, a task heavily relying on the backbone architecture for multimodal representation and temporal state tracking\. Despite recent progress, mainstream frameworks like AV\-WaN\[[5](https://arxiv.org/html/2607.13110#bib.bib1),[6](https://arxiv.org/html/2607.13110#bib.bib8)\]still suffer from representation mismatches due to traditional paradigms: their gated recurrent units utilize non\-selective state updates that dilute critical acoustic signals with redundant environmental observations, while their convolutional neural networks introduce a local bias that disrupts the global time\-frequency topological dependencies of spectrograms\. These backbone\-level representational limitations have become a substantial bottleneck restricting further breakthroughs in navigation performance\.

To address the above limitations, and inspired by the outstanding performance of structured state\-space models\[[10](https://arxiv.org/html/2607.13110#bib.bib37)\]in sequence modeling, this paper constructs a hybrid state\-space architecture named Samba\. This framework establishes a heterogeneous hybrid modeling paradigm driven by Mamba\[[10](https://arxiv.org/html/2607.13110#bib.bib37)\], aiming to enhance the system’s representation capability by introducing a data\-adaptive selective scanning mechanism\. We design the Mamba State Encoder to replace traditional gated recurrent units, enabling accurate extraction and preservation of critical navigation states within complex sequences\. Meanwhile, we build a bidirectional Audio Mamba Encoder, whose global modeling property resolves the time\-frequency fragmentation issue in spectrogram analysis encountered by conventional convolutional operators\. Furthermore, this framework strategically retains convolutional architectures in the visual branch to leverage the advantages of spatial inductive bias\. Ultimately, efficient interpretation of multimodal signals is achieved through Mamba’s linear complexity and content\-aware characteristics\.

In summary, our main contributions are as follows:

- •We establish a hybrid state\-space architecture named Samba, representing the first attempt to introduce state\-space models as a backbone network in the field of audio\-visual navigation\.
- •We design an adaptive selection\-aware Mamba State Encoder and a bidirectional Audio Mamba Encoder, which address the representation dilution issue in state modeling and the structural deficiency in acoustic features, respectively\.
- •Extensive experiments validate that Samba achieves significant improvements in navigation success rate while maintaining parameter efficiency, outperforming all existing state\-of\-the\-art baseline models\.

## IIRelated Work

![Refer to caption](https://arxiv.org/html/2607.13110v1/x1.png)Figure 1:Overall Architecture of the Samba Framework\.### II\-AAudio\-Visual Navigation

The AVN task\[[5](https://arxiv.org/html/2607.13110#bib.bib1),[9](https://arxiv.org/html/2607.13110#bib.bib2)\]requires an agent to localize sound sources in complex 3D environments by jointly analyzing visual observations and binaural audio signals\. Early research mainly focused on geometry\-driven navigation strategies\[[2](https://arxiv.org/html/2607.13110#bib.bib35)\], while in recent years, the end\-to\-end paradigm based on deep reinforcement learning has become mainstream in this field\. Within this paradigm, milestone works such as AV\-WaN\[[6](https://arxiv.org/html/2607.13110#bib.bib8),[26](https://arxiv.org/html/2607.13110#bib.bib17),[30](https://arxiv.org/html/2607.13110#bib.bib32),[28](https://arxiv.org/html/2607.13110#bib.bib10),[13](https://arxiv.org/html/2607.13110#bib.bib22),[7](https://arxiv.org/html/2607.13110#bib.bib20),[11](https://arxiv.org/html/2607.13110#bib.bib29),[27](https://arxiv.org/html/2607.13110#bib.bib18)\]have established a standard backbone architecture that fuses audio, vision, and historical states\. However, although these methods have achieved progress on standard simulation platforms such as SoundSpaces\[[5](https://arxiv.org/html/2607.13110#bib.bib1),[17](https://arxiv.org/html/2607.13110#bib.bib5),[19](https://arxiv.org/html/2607.13110#bib.bib6)\], their underlying architectures often suffer from a trade\-off between representation efficiency and computational overhead\. Most existing studies\[[22](https://arxiv.org/html/2607.13110#bib.bib36),[25](https://arxiv.org/html/2607.13110#bib.bib13),[12](https://arxiv.org/html/2607.13110#bib.bib48)\]focus on improving multimodal fusion mechanisms\. In contrast, this work goes further and attempts to enhance the perception and reasoning efficiency of the navigation system by reshaping the backbone network at a fundamental level\[[29](https://arxiv.org/html/2607.13110#bib.bib40),[31](https://arxiv.org/html/2607.13110#bib.bib41),[24](https://arxiv.org/html/2607.13110#bib.bib42),[32](https://arxiv.org/html/2607.13110#bib.bib44),[21](https://arxiv.org/html/2607.13110#bib.bib43),[3](https://arxiv.org/html/2607.13110#bib.bib45),[8](https://arxiv.org/html/2607.13110#bib.bib46),[14](https://arxiv.org/html/2607.13110#bib.bib47),[34](https://arxiv.org/html/2607.13110#bib.bib49),[33](https://arxiv.org/html/2607.13110#bib.bib50)\]\.

### II\-BState Space Models and Mamba

State Space Models \(SSMs\)\[[10](https://arxiv.org/html/2607.13110#bib.bib37)\], as an emerging deep learning architecture, are gradually becoming a strong alternative to Recurrent Neural Networks \(RNNs\)\[[15](https://arxiv.org/html/2607.13110#bib.bib39)\]and Transformers\[[20](https://arxiv.org/html/2607.13110#bib.bib14)\]due to their superior performance in handling extremely long\-sequence tasks\. From the early Structured State Space model\[[10](https://arxiv.org/html/2607.13110#bib.bib37)\]to the recent Mamba\[[10](https://arxiv.org/html/2607.13110#bib.bib37)\]architecture, such models have successfully captured complex sequential dependencies while maintaining linear time complexity by introducing the Selective Scan Mechanism\. In the fields of embodied intelligence and computer vision, Mamba\[[10](https://arxiv.org/html/2607.13110#bib.bib37)\]has demonstrated the potential to outperform traditional architectures, enabling efficient autoregressive inference with extremely low memory overhead\. The research motivation of this paper stems precisely from this: to address the long\-standing state aggregation bottleneck in navigation tasks by leveraging the hardware\-aware efficiency of Mamba\[[10](https://arxiv.org/html/2607.13110#bib.bib37)\]\.

## IIIProposed Method

### III\-AFramework Overview

Samba is an end\-to\-end system designed specifically for audio\-visual navigation, which restructures perceptual encoding and state modeling operators using hierarchical selective state spaces\. To ensure seamless integration in heterogeneous environments and flexibility in policy sampling, the underlying system employs a lightweight Mamba\[[10](https://arxiv.org/html/2607.13110#bib.bib37)\]operator library built upon standard PyTorch\[[16](https://arxiv.org/html/2607.13110#bib.bib38)\]operators\. As shown in Figure[1](https://arxiv.org/html/2607.13110#S2.F1), at navigation steptt, the agent takes a depth map, binaural spectrogram, and global geometric mapGtG\_\{t\}as inputs\. In the perception layer, the Audio Mamba Encoder\(AME\) extracts acoustic featuresbtb\_\{t\}with selective context through tokenization, which are then fed into the Mamba State Encoder \(M\-SE\) together with the geometric vectorgtg\_\{t\}derived from the depth map and the acoustic map featureata\_\{t\}\. Leveraging linear recurrence, M\-SE updates the historical stateht−1h\_\{t\-1\}to the current global statehth\_\{t\}, enabling high\-fidelity modeling of long\-range environmental characteristics\. Subsequently, the Actor\-Critic\-based prediction network uses the PPO algorithm to sample non\-myopic waypointswt=\(Δ​x,Δ​y\)w\_\{t\}=\(\\Delta x,\\Delta y\)over the action mapWtW\_\{t\}\. Finally, these waypoints drive the path planner to guide the agent to accurately reach the target sound source in unknown environments\.

![Refer to caption](https://arxiv.org/html/2607.13110v1/x2.png)Figure 2:Architectural Comparison between the baseline GRU and our proposed Mamba State Encoder \(M\-SE\)\. The left side illustrates the classic Gated Recurrent Unit, while the right side details the Audio Mamba State Encoder designed in this paper\.
### III\-BAudio Mamba Encoder

Algorithm 1Architectural Procedure of Audio Mamba Encoder0:Audio spectrogram

St∈ℝB×C×F×TS\_\{t\}\\in\\mathbb\{R\}^\{B\\times C\\times F\\times T\}
0:Audio state vector

bt∈ℝdo​u​tb\_\{t\}\\in\\mathbb\{R\}^\{d\_\{out\}\}
1:

B,C,F,T←shape of​StB,C,F,T\\leftarrow\\text\{shape of \}S\_\{t\}
2:

x←Reshape​\(Permute​\(St,\(0,3,2,1\)\),\(B,T,C⋅F\)\)x\\leftarrow\\text\{Reshape\}\(\\text\{Permute\}\(S\_\{t\},\(0,3,2,1\)\),\(B,T,C\\cdot F\)\)\{Tokenization and spectral flattening\}

3:

x←ReLU​\(LayerNorm​\(𝐖p​x\)\)x\\leftarrow\\text\{ReLU\}\(\\text\{LayerNorm\}\(\\mathbf\{W\}\_\{p\}x\)\)\{Project features to

dm​o​d​e​ld\_\{model\}\}

4:

x←x\+𝐏\[:,:T,:\]x\\leftarrow x\+\\mathbf\{P\}\[:,:T,:\]\{Temporal positional encoding injection\}

5:for

i=0i=0to

Nl​a​y​e​r​s−1N\_\{layers\}\-1do

6:if

i\(mod2\)==1i\\pmod\{2\}==1then

7:

x←Flip​\(x,dim=1\)x\\leftarrow\\text\{Flip\}\(x,\\text\{dim\}=1\)\{Sequence reversal\}

8:

x←MambaLayeri​\(x\)x\\leftarrow\\text\{MambaLayer\}\_\{i\}\(x\)\{Backward selective scanning\}

9:

x←Flip​\(x,dim=1\)x\\leftarrow\\text\{Flip\}\(x,\\text\{dim\}=1\)\{Restore temporal order\}

10:else

11:

x←MambaLayeri​\(x\)x\\leftarrow\\text\{MambaLayer\}\_\{i\}\(x\)\{Forward selective scanning\}

12:endif

13:endfor

14:

fb←Mean​\(LayerNorm​\(x\),dim=1\)f\_\{b\}\\leftarrow\\text\{Mean\}\(\\text\{LayerNorm\}\(x\),\\text\{dim\}=1\)\{Global aggregate pooling\}

15:

bt←𝐖h​fbb\_\{t\}\\leftarrow\\mathbf\{W\}\_\{h\}f\_\{b\}\{Final audio state projection\}

16:return

btb\_\{t\}

To address the lack of global context caused by limited local receptive fields in traditional convolutional architectures when processing complex acoustic observations, we design an Audio Mamba Encoder \(AME\) based on selective state\-space models\. The core logic of this module is to elevate the audio feature extraction process from local operator transformations to global selective scanning, thereby accurately capturing the cross\-time\-frequency directional characteristics embedded in the binaural spectrogramSt∈ℝC×F×TS\_\{t\}\\in\\mathbb\{R\}^\{C\\times F\\times T\}\. As shown in Algorithm 1, in the feature tokenization stage, the system first linearly merges the channel and frequency dimensions of the original spectrogram via dimension permutation and rearrangement to construct a discrete token sequence\. Specifically, for each time stepτ∈\{1,…,T\}\\tau\\in\\\{1,\\dots,T\\\}, we flatten its cross\-dimensional features and project them into the hidden space, yielding the initial feature vectorxτx\_\{\\tau\}as:

xτ=ReLU⁡\(LN⁡\(𝐖p⋅\[S1,1,τ,…,SC,F,τ\]⊤\+𝐛p\)\),x\_\{\\tau\}=\\operatorname\{ReLU\}\\bigl\(\\operatorname\{LN\}\(\\mathbf\{W\}\_\{p\}\\cdot\[S\_\{1,1,\\tau\},\\dots,S\_\{C,F,\\tau\}\]^\{\\top\}\+\\mathbf\{b\}\_\{p\}\)\\bigr\),\(1\)where𝐖p∈ℝdm​o​d​e​l×\(C×F\)\\mathbf\{W\}\_\{p\}\\in\\mathbb\{R\}^\{d\_\{model\}\\times\(C\\times F\)\}denotes the learnable linear projection matrix\. To compensate for the temporal position information that may be lost when state\-space models process sequences, we introduce a positional embedding matrix𝐏\\mathbf\{P\}and convert the projected sequence into a hidden input with temporal features𝐳τ=xτ\+𝐏τ\\mathbf\{z\}\_\{\\tau\}=x\_\{\\tau\}\+\\mathbf\{P\}\_\{\\tau\}\.

The inference logic of AME further leverages a bidirectional selective scan mechanism to enhance representations\. Given that the interaural time difference and interaural level difference, on which sound source localization relies, exhibit strong temporal sensitivity, we abandon traditional static convolutional operators and adopt a heterogeneous bidirectional Mamba architecture instead\. At each operator layerii, the model models long\-range dependencies in both forward and backward directions by alternately flipping the time axis of the sequence, whose layer\-wise output𝐲i\\mathbf\{y\}\_\{i\}can be formulated as:

𝐲i=\{MambaLayeri⁡\(𝐳\)if​i​is evenFlip⁡\(MambaLayeri⁡\(Flip⁡\(𝐳\)\)\)if​i​is odd\\mathbf\{y\}\_\{i\}=\\begin\{cases\}\\operatorname\{MambaLayer\}\_\{i\}\(\\mathbf\{z\}\)&\\text\{if \}i\\text\{ is even\}\\\\ \\operatorname\{Flip\}\\bigl\(\\operatorname\{MambaLayer\}\_\{i\}\(\\operatorname\{Flip\}\(\\mathbf\{z\}\)\)\\bigr\)&\\text\{if \}i\\text\{ is odd\}\\end\{cases\}\(2\)
This design ensures that the model can simultaneously parse the causal and anticausal correlations of audio signals in the time domain, achieving high\-fidelity analysis of the sound field in complex static indoor scenes\. Compared with parameter\-redundant CNNs\[[23](https://arxiv.org/html/2607.13110#bib.bib34)\], AME can adaptively adjust the representation weights according to the intensity distribution of the current acoustic input\. Finally, the sequence features after global average pooling are mapped into the audio state vectorbtb\_\{t\}by a linear head, thereby providing high\-quality acoustic semantic support for the subsequent global state aggregation\.

### III\-CMamba State Encoder

TABLE I:Performance comparison with other methods under the Depth setting\. SPL, SR, SNA are percentages\.We reformulate the state modeling operator of the backbone network into the Mamba State Encoder \(M\-SE\) based on selective state\-space models, marking a complete shift from the traditional gated recurrent logic to a modern sequential modeling paradigm\. As illustrated in Fig\. 3, the M\-SE takes multimodal fused featuresxt∈ℝdm​o​d​e​lx\_\{t\}\\in\\mathbb\{R\}^\{d\_\{model\}\}from the perception layer and achieves real\-time aggregation of navigation states in a fully data\-driven manner\. Before feeding into the core state\-space operator, the input signal is first processed by a combination of linear projection and local 1D convolution, which effectively preserves local spatial correlations and feature smoothness during temporal sequence progression\. Compared with the relatively fixed gating update mechanism of GRUs, the key advantage of M\-SE lies in its selective mechanism: the parametersΔ\\Delta,𝐁\\mathbf\{B\}, and𝐂\\mathbf\{C\}are no longer static weights, but functions dynamically generated conditioned on the current inputxtx\_\{t\}\.

In terms of mathematical modeling, the underlying logic of M\-SE originates from the discretization of continuous state\-space equations\. For the continuous hidden stateh​\(t\)h\(t\), its evolution is governed by the linear differential equationh˙​\(t\)=𝐀​h​\(t\)\+𝐁​x​\(t\)\\dot\{h\}\(t\)=\\mathbf\{A\}h\(t\)\+\\mathbf\{B\}x\(t\)\. To handle discrete navigation steps computationally, we apply the zero\-order hold rule for discretization, thereby deriving the explicit expressions of the discrete transition matrix𝐀¯\\mathbf\{\\bar\{A\}\}and input matrix𝐁¯\\mathbf\{\\bar\{B\}\}:

𝐀¯\\displaystyle\\bar\{\\mathbf\{A\}\}=exp⁡\(Δ​𝐀\),\\displaystyle=\\exp\(\\Delta\\mathbf\{A\}\),\(3\)𝐁¯\\displaystyle\\bar\{\\mathbf\{B\}\}=\(Δ​𝐀\)−1​\(exp⁡\(Δ​𝐀\)−𝐈\)⋅Δ​𝐁\.\\displaystyle=\(\\Delta\\mathbf\{A\}\)^\{\-1\}\\bigl\(\\exp\(\\Delta\\mathbf\{A\}\)\-\\mathbf\{I\}\\bigr\)\\cdot\\Delta\\mathbf\{B\}\.\(4\)
In our practical lightweight PyTorch implementation, the above discretization procedure is further simplified to𝐀¯=exp⁡\(Δ​𝐀\)\\mathbf\{\\bar\{A\}\}=\\exp\(\\Delta\\mathbf\{A\}\)and𝐁¯=Δ​𝐁\\mathbf\{\\bar\{B\}\}=\\Delta\\mathbf\{B\}\. Based on this, the recurrent update equation for the hidden state𝐡t∈ℝdi​n​n​e​r×ds​t​a​t​e\\mathbf\{h\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{inner\}\\times d\_\{state\}\}can be formulated as:

𝐡t=𝐀¯​𝐡t−1\+𝐁¯​xt,\\mathbf\{h\}\_\{t\}=\\bar\{\\mathbf\{A\}\}\\,\\mathbf\{h\}\_\{t\-1\}\+\\bar\{\\mathbf\{B\}\}\\,x\_\{t\},\(5\)this design endows the agent with stronger perceptual modeling capabilities from a mathematical essence\. When the agent faces the challenge of information representation dilution during exploration, M\-SE can dynamically control the decay rate of historical memory by adjusting the step sizeΔ\\Delta, thereby focusing on acoustic landmarks that are critical to target orientation\. The computed hidden state is feature\-mapped via the output parameter𝐂\\mathbf\{C\}, then fused with the residual path through a gating mechanism\. The final global hidden statehth\_\{t\}serves as the core input to the decision\-making system\.

To balance the large\-scale parallelism of policy training and the real‑time response requirements of embodied navigation, M‑SE supports dual‑mode switching between parallel and recurrent execution in its architecture\. During training, the system employs a parallel scan algorithm for efficient gradient computation over long observation sequences, while during deployment, it seamlessly switches to single‑step recurrent mode to guarantee extremely low latency in decision‑making\.

In the policy learning process, the system dynamically maintains the internal state matrices and local convolution buffers of the SSM through a specially designed state unpacking and packing mechanism\. Particularly critical for adapting to the sampling nature of reinforcement learning, we introduce masking logic that forcibly resets Mamba’s internal hidden state at the end of each episode\. This effectively prevents state contamination from prior experience on new tasks and ensures the stability of PPO optimization\. Through such a comprehensive upgrade from operators to the underlying state machine, M‑SE provides Samba with far greater long‑range modeling depth than traditional recurrent architectures, significantly enhancing the agent’s perceptual robustness in complex and unknown environments\.

## IVExperiments

The experimental evaluation of this study mainly relies on the SoundSpaces acoustic simulation platform constructed on the Habitat simulator, and incorporates two widely adopted public datasets in the field of embodied intelligence namely Replica\[[18](https://arxiv.org/html/2607.13110#bib.bib4)\]and Matterport3D\(MP3D\)\[[4](https://arxiv.org/html/2607.13110#bib.bib3)\]\. The MP3D dataset covers 85 real\-world indoor environments dominated by residential spaces, providing the agent with complete 3D mesh models and corresponding panoramic image scanning data\. The Replica\[[18](https://arxiv.org/html/2607.13110#bib.bib4)\]dataset contains 18 synthetic indoor scenes including apartments and hotels, which outperforms MP3D\[[4](https://arxiv.org/html/2607.13110#bib.bib3)\]in both the precision of geometric details and the realism of lighting effects\.

To reconstruct complex acoustic propagation characteristics in virtual scenes, the SoundSpaces platform synthesizes physically realistic sound sources in real time by convolving selected audio signals with binaural room impulse responses at corresponding spatial positions\. In the specific experimental setup, we adopt 102 distinct natural sounds as source stimuli to ensure that the model has sufficient generalization ability when facing diverse acoustic inputs\. Through such a high\-fidelity multimodal simulation environment, we are able to conduct a comprehensive and rigorous validation of the navigation performance of the Samba architecture under complex physical constraints\.

In our experiments, two different sound source conditions are included\. \(1\) Heard: The target sound source is a telephone ringtone, which is used in the training, validation, and test sets\. \(2\) Unheard: The 102 sound sources are divided into three non\-overlapping groups\. Specifically, 78 sound sources are used for training scenes, 11 for validation scenes, and the remaining 18 for test scenes\[[22](https://arxiv.org/html/2607.13110#bib.bib36)\]\. All test scenes are unseen during training\[[22](https://arxiv.org/html/2607.13110#bib.bib36)\]\.

### IV\-AEvaluation Metrics

Following established protocols, the navigation performance is quantified via Success Rate \(S​RSR\), Success weighted by Path Length \(S​P​LSPL\), and Success Navigation Accuracy \(S​N​ASNA\)\.S​RSRcomputes the fraction of runs where the agent successfully stops at the target\. Trajectory optimality is captured byS​P​LSPLby balancing the shortest pathlil\_\{i\}against the execution pathpip\_\{i\}, whereas strategic precision is monitored byS​N​ASNA, which uses the action countaia\_\{i\}to suppress redundant maneuvers like in\-place rotations\. We formulate these definitions as follows:

SR=1N​∑i=1NSi,\\text\{SR\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}S\_\{i\},\(6\)SPL=1N​∑i=1NSi⋅limax⁡\(pi,li\)​\[[6](https://arxiv.org/html/2607.13110#bib.bib8)\],\\text\{SPL\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}S\_\{i\}\\cdot\\frac\{l\_\{i\}\}\{\\max\(p\_\{i\},l\_\{i\}\)\}~\\cite\[cite\]\{\[\\@@bibref\{\}\{Waypoint\}\{\}\{\}\]\},\(7\)SNA=1N​∑i=1NSi⋅liai​\[[1](https://arxiv.org/html/2607.13110#bib.bib11)\],\\text\{SNA\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}S\_\{i\}\\cdot\\frac\{l\_\{i\}\}\{a\_\{i\}\}~\\cite\[cite\]\{\[\\@@bibref\{\}\{SPL\}\{\}\{\}\]\},\(8\)In these expressions,NNdenotes the total number of test episodes whileSi∈\{0,1\}S\_\{i\}\\in\\\{0,1\\\}serves as the binary success indicator for theii\-th episode\. The variableslil\_\{i\}andpip\_\{i\}correspond to the shortest and actual path lengths respectively, andaia\_\{i\}records the total count of actions executed throughout the navigation process\.

### IV\-BQuantitative Experimental Results

As summarized in Table[I](https://arxiv.org/html/2607.13110#S3.T1), Samba consistently outperforms all existing baselines across all test environments\. On the Replica dataset\[[18](https://arxiv.org/html/2607.13110#bib.bib4)\], for instance, it achieves a 72\.8% success rate under the unheard\-sound, unseen\-scene setting, representing a 20 percentage point absolute improvement over the prior state\-of\-the\-art method AV\-WaN\[[6](https://arxiv.org/html/2607.13110#bib.bib8)\]\. We also observe notable improvements in trajectory and decision efficiency: SPL climbs from 34\.7% to 42\.4%, and SNA grows from 27\.1% to 30\.8%\. These simultaneous gains suggest that our selective scanning mechanism supports robust state tracking and extracts precise directional features from dynamic observation streams, which effectively reduces aimless wandering during navigation\.

The framework also adapts well to the more acoustically complex MP3D dataset\[[4](https://arxiv.org/html/2607.13110#bib.bib3)\]\. When tested with unheard sound sources, Samba achieves a 68\.0% SR, 47\.1% SPL, and 36\.2% SNA\. Unlike conventional recurrent architectures that suffer from state dilution over long sequences, Samba maintains a better balance between audio and visual inputs\. This result confirms that our redesigned backbone brings critical structural advantages for large\-scale embodied navigation tasks\.

### IV\-CEfficiency Analysis

TABLE II:Quantitative comparison and analysis between the Samba architecture and the AV\-WaN baseline model in terms of audio encoding modules, state encoding modules, and total parameter sizes across different datasets\. Lower values indicate improved model efficiency\.Table[II](https://arxiv.org/html/2607.13110#S4.T2)confirms that Samba achieves strong structural efficiency through its core operator design\. The audio encoder contains only0\.4​M0\.4\\text\{M\}parameters, a43%43\\%reduction from the0\.7​M0\.7\\text\{M\}baseline, which demonstrates that a compact selective scanning mechanism captures global acoustic features more effectively than standard convolutional layers\. We similarly trim state encoding parameters from3\.9​M3\.9\\text\{M\}to2\.7​M2\.7\\text\{M\}to cut architectural redundancy and maintain stable training when data is limited\. Total parameter counts differ across datasets due to varying sensor resolutions, reaching10\.5​M10\.5\\text\{M\}on Replica and4\.6​M4\.6\\text\{M\}on MP3D, yet Samba consistently outperforms AV\-WaN in both settings\. Overall, our redesigned backbone preserves critical environmental cues without redundant network stacking, and adapts the framework well for edge robotics deployment\.

### IV\-DAblation Studies

Ablation results in Table[III](https://arxiv.org/html/2607.13110#S4.T3)and Table[IV](https://arxiv.org/html/2607.13110#S4.T4)evaluate the incremental gains of our Mamba\-based components, where “w/o AME” and “w/o M\-SE” represent reverting the respective modules back to the baseline CNN and GRU architectures\. Replacing AME triggers a substantial performance collapse, withS​RSRfor unheard sounds dropping from72\.8%72\.8\\%to59\.0%59\.0\\%on Replica and from68\.0%68\.0\\%to62\.6%62\.6\\%on MP3D\. Conversely, the configuration without M\-SE results in relatively moderate fluctuations, with ReplicaS​RSRmaintaining71\.0%71\.0\\%\. These findings confirm that perceptual precision is the dominant driver of success, validating our perception\-centric architectural design\.

TABLE III:Ablation study on the Fusion Controller of CRFN on the Replica dataset\.TABLE IV:Ablation study on the Fusion Controller of CRFN on the Matterport3D dataset\.

## VCONCLUSIONS

We propose Samba, a hybrid architecture built on state\-space models, to overcome the long\-standing efficiency and modeling bottlenecks in audio\-visual navigation\. Samba integrates an adaptive selective Mamba State Encoder with a bidirectional Audio Mamba Encoder; this combination successfully captures global time\-frequency correlations where convolutions fail and prevents the representation dilution typical of recurrent units\. Our empirical validation shows that updating the backbone not only yields stronger embodied representations but also slashes parameter size and computation\. This shift provides an efficient, high\-performance reference design for future embodied AI deployment\.

## References

- \[1\]P\. Anderson, A\. Chang, D\. S\. Chaplot, A\. Dosovitskiy, S\. Gupta, V\. Koltun, J\. Kosecka, J\. Malik, R\. Mottaghi, M\. Savva,et al\.\(2018\)On evaluation of embodied navigation agents\.arXiv preprint arXiv:1807\.06757\.Cited by:[8](https://arxiv.org/html/2607.13110#S4.E8.m1.1.1.1.1.3.3.2.3)\.
- \[2\]\(2015\)A survey on sound source localization in robotics: from binaural to array processing methods\.Computer Speech & Language34\(1\),pp\. 87–112\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[3\]Y\. Cao, Y\. Li, L\. Wang, and Y\. Yu\(2024\)VNet: a gan\-based multi\-tier discriminator network for speech synthesis vocoders\.In2024 IEEE International Conference on Systems, Man, and Cybernetics \(SMC\),pp\. 4384–4389\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[4\]A\. Chang, A\. Dai, T\. Funkhouser, M\. Halber, M\. Niebner, M\. Savva, S\. Song, A\. Zeng, and Y\. Zhang\(2018\)Matterport3D: learning from rgb\-d data in indoor environments\.In7th IEEE International Conference on 3D Vision, 3DV 2017,pp\. 667–676\.Cited by:[§IV\-B](https://arxiv.org/html/2607.13110#S4.SS2.p2.1),[§IV](https://arxiv.org/html/2607.13110#S4.p1.1)\.
- \[5\]C\. Chen, U\. Jain, C\. Schissler, S\. V\. A\. Gari, Z\. Al\-Halah, V\. K\. Ithapu, P\. Robinson, and K\. Grauman\(2020\)SoundSpaces: audio\-visual navigation in 3d environments\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 17–36\.Cited by:[§I](https://arxiv.org/html/2607.13110#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1),[TABLE I](https://arxiv.org/html/2607.13110#S3.T1.12.20.6.1)\.
- \[6\]C\. Chen, S\. Majumder, Z\. Al\-Halah, R\. Gao, S\. K\. Ramakrishnan, and K\. Grauman\(2021\-May 3–7\)Learning to set waypoints for audio\-visual navigation\.In9th International Conference on Learning Representations, ICLR 2021,Virtual Event, Austria\.Cited by:[§I](https://arxiv.org/html/2607.13110#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1),[TABLE I](https://arxiv.org/html/2607.13110#S3.T1.12.15.1.1),[TABLE I](https://arxiv.org/html/2607.13110#S3.T1.12.16.2.1),[TABLE I](https://arxiv.org/html/2607.13110#S3.T1.12.17.3.1),[TABLE I](https://arxiv.org/html/2607.13110#S3.T1.12.18.4.1),[TABLE I](https://arxiv.org/html/2607.13110#S3.T1.12.22.8.1),[7](https://arxiv.org/html/2607.13110#S4.E7.m1.4.4.1.1.3.3.2.3),[§IV\-B](https://arxiv.org/html/2607.13110#S4.SS2.p1.1),[TABLE II](https://arxiv.org/html/2607.13110#S4.T2.3.5.1.1)\.
- \[7\]J\. Chen, W\. Wang, S\. Liu, H\. Li, and Y\. Yang\(2023\)Omnidirectional information gathering for knowledge transfer\-based audio\-visual navigation\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 10993–11003\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[8\]J\. Fu, Y\. Yu, and L\. Wang\(2025\)FSDENet: a frequency and spatial domains based detail enhancement network for remote sensing semantic segmentation\.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[9\]C\. Gan, Y\. Zhang, J\. Wu, B\. Gong, and J\. B\. Tenenbaum\(2020\)Look, listen, and act: towards audio\-visual embodied navigation\.In2020 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 9701–9707\.Cited by:[§I](https://arxiv.org/html/2607.13110#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1),[TABLE I](https://arxiv.org/html/2607.13110#S3.T1.12.19.5.1)\.
- \[10\]A\. Gu and T\. Dao\(2024\)Mamba: linear\-time sequence modeling with selective state spaces\.InFirst conference on language modeling,Cited by:[§I](https://arxiv.org/html/2607.13110#S1.p2.1),[§II\-B](https://arxiv.org/html/2607.13110#S2.SS2.p1.1),[§III\-A](https://arxiv.org/html/2607.13110#S3.SS1.p1.9)\.
- \[11\]E\. Kolve, R\. Mottaghi, W\. Han, E\. VanderBilt, L\. Weihs, A\. Herrasti, M\. Deitke, K\. Ehsani, D\. Gordon, Y\. Zhu,et al\.\(2017\)Ai2\-thor: an interactive 3d environment for visual ai\.arXiv preprint arXiv:1712\.05474\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[12\]J\. Li, Y\. Yu, L\. Wang, F\. Sun, and W\. Zheng\(2025\)Audio\-guided dynamic modality fusion with stereo\-aware attention for audio\-visual navigation\.InInternational Conference on Neural Information Processing,pp\. 346–359\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[13\]X\. Liu, S\. Paul, M\. Chatterjee, and A\. Cherian\(2024\)Caven: an embodied conversational agent for efficient audio\-visual navigation in noisy environments\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 3765–3773\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[14\]A\. Mattursun, L\. Wang, and Y\. Yu\(2024\)BSS\-cffma: cross\-domain feature fusion and multi\-attention speech enhancement network based on self\-supervised embedding\.In2024 IEEE International Conference on Systems, Man, and Cybernetics \(SMC\),pp\. 3589–3594\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[15\]L\. R\. Medsker, L\. Jain,et al\.\(2001\)Recurrent neural networks\.Design and applications5\(64\-67\),pp\. 2\.Cited by:[§II\-B](https://arxiv.org/html/2607.13110#S2.SS2.p1.1)\.
- \[16\]A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga,et al\.\(2019\)Pytorch: an imperative style, high\-performance deep learning library\.Advances in neural information processing systems32\.Cited by:[§III\-A](https://arxiv.org/html/2607.13110#S3.SS1.p1.9)\.
- \[17\]M\. Savva, A\. Kadian, O\. Maksymets, Y\. Zhao, E\. Wijmans, B\. Jain, J\. Straub, J\. Liu, V\. Koltun, J\. Malik,et al\.\(2019\)Habitat: a platform for embodied ai research\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 9339–9347\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[18\]J\. Straub, T\. Whelan, L\. Ma, Y\. Chen, E\. Wijmans, S\. Green, J\. J\. Engel, R\. Mur\-Artal, C\. Ren, S\. Verma,et al\.\(2019\)The replica dataset: a digital replica of indoor spaces\.arXiv preprint arXiv:1906\.05797\.Cited by:[§IV\-B](https://arxiv.org/html/2607.13110#S4.SS2.p1.1),[§IV](https://arxiv.org/html/2607.13110#S4.p1.1)\.
- \[19\]A\. Szot, A\. Clegg, E\. Undersander, E\. Wijmans, Y\. Zhao, J\. Turner, N\. Maestre, M\. Mukadam, D\. S\. Chaplot, O\. Maksymets,et al\.\(2021\)Habitat 2\.0: training home assistants to rearrange their habitat\.Advances in neural information processing systems34,pp\. 251–266\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[20\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§II\-B](https://arxiv.org/html/2607.13110#S2.SS2.p1.1)\.
- \[21\]X\. Wang, L\. Wang, Y\. Yu, and X\. Jiao\(2025\)Modality\-invariant bidirectional temporal representation distillation network for missing multimodal sentiment analysis\.InICASSP 2025\-2025 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[22\]Y\. Wang, Y\. Yu, and B\. Ren\(2026\)Residual cross\-modal fusion networks for audio\-visual navigation\.arXiv preprint arXiv:2601\.08868\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1),[TABLE I](https://arxiv.org/html/2607.13110#S3.T1.12.21.7.1),[§IV](https://arxiv.org/html/2607.13110#S4.p3.1)\.
- \[23\]J\. Wu\(2017\)Introduction to convolutional neural networks\.National Key Lab for Novel Software Technology\. Nanjing University\. China5\(23\),pp\. 495\.Cited by:[§III\-B](https://arxiv.org/html/2607.13110#S3.SS2.p5.1)\.
- \[24\]D\. Yang, Y\. Yu, and L\. Wang\(2026\)Beyond textual knowledge: leveraging multimodal knowledge bases for enhancing vision\-and\-language navigation\.Information Processing & Management63\(6\),pp\. 104766\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[25\]Y\. Yu, L\. Cao, F\. Sun, X\. Liu, and L\. Wang\(2022\-11\)Pay self\-attention to audio\-visual navigation\.In33rd British Machine Vision Conference 2022, BMVC 2022,London, UK,pp\. 46\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[26\]Y\. Yu, L\. Cao, F\. Sun, C\. Yang, H\. Lai, and W\. Huang\(2023\-04\)Echo\-enhanced embodied visual navigation\.Neural Computation35\(5\),pp\. 958–976\.External Links:ISSN 0899\-7667Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[27\]Y\. Yu, C\. Chen, L\. Cao, F\. Yang, and F\. Sun\(2023\-08\)Measuring acoustics with collaborative multiple agents\.InProceedings of the 32nd International Joint Conference on Artificial Intelligence \(IJCAI\-23\),Macao, China\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[28\]Y\. Yu, W\. Huang, F\. Sun, C\. Chen, Y\. Wang, and X\. Liu\(2022\)Sound adversarial audio\-visual navigation\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[29\]Y\. Yu and S\. Sun\(2025\)DGFNet: end\-to\-end audio\-visual source separation based on dynamic gating fusion\.InProceedings of the 2025 International Conference on Multimedia Retrieval,pp\. 1730–1738\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[30\]Y\. Yu and D\. Yang\(2025\)DOPE: dual object perception\-enhancement network for vision\-and\-language navigation\.InProceedings of the 2025 International Conference on Multimedia Retrieval,pp\. 1739–1748\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[31\]Y\. Yu, H\. Zhang, and M\. Zhu\(2025\)Dynamic multi\-target fusion for efficient audio\-visual navigation\.arXiv preprint arXiv:2509\.21377\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[32\]H\. Zhang, Y\. Yu, L\. Wang, F\. Sun, and W\. Zheng\(2025\)Advancing audio\-visual navigation through multi\-agent collaboration in 3d environments\.InInternational Conference on Neural Information Processing,pp\. 502–516\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[33\]H\. Zhang, Y\. Yu, L\. Wang, F\. Sun, and W\. Zheng\(2025\)Iterative residual cross\-attention mechanism: an integrated approach for audio\-visual navigation tasks\.arXiv preprint arXiv:2509\.25652\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.
- \[34\]J\. Zhang, L\. Wang, Y\. Yu, and M\. Xu\(2024\)Nonlinear regularization decoding method for speech recognition\.Sensors24\(12\),pp\. 3846\.Cited by:[§II\-A](https://arxiv.org/html/2607.13110#S2.SS1.p1.1)\.