Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings

# Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings
Source: [https://arxiv.org/html/2607.12071](https://arxiv.org/html/2607.12071)
11institutetext:Center for BioMed\-X Research, Academy for Advanced Interdisciplinary Studies,Peking University, Beijing, China22institutetext:Speech and Hearing Research Center, School of Intelligence Science and Technology,Peking University, Beijing, China33institutetext:State Key Laboratory of General Artificial Intelligence, Beijing, China44institutetext:National Biomedical Imaging Center, State Key Laboratory of Membrane Biology, Institute of Molecular Medicine, Peking\-Tsinghua Center for Life Sciences, College of Future Technology, Peking University, Beijing, China
44email:janechenjing@pku\.edu\.cn###### Abstract

Continuous semantic reconstruction from non\-invasive neural recordings remains limited by the representational mismatch between semantic feature spaces and neural coding patterns, which severely impedes cross\-modal alignment between high\-noise neural signals and target semantic features\. Prior semantic decoders have predominantly relied on static lexical representations or dynamic contextualized representations in isolation\. This single\-feature approach inevitably leads to severe information loss, as it fails to account for the human brain’s capacity to integrate stable word attributes and dynamic contexts simultaneously\.

To bridge this gap, this study introduces a multi\-feature fusion framework for non\-invasive semantic reconstruction, systematically benchmarking two integration approaches: linear Naive Concatenation and non\-linear Multi\-Head Cross\-Attention\. Within this framework, our approach complements static lexical representations \(W2V\) with dynamic contextual representations \(GPT\) via an interactive gating mechanism to facilitate cooperative processing during language comprehension\.

Evaluated through extensive semantic reconstruction and text generation experiments, our framework reveals a robust performance hierarchy:Cross\-Att\>Concat\>GPT\>W2V\\text\{Cross\-Att\}\>\\text\{Concat\}\>\\text\{GPT\}\>\\text\{W2V\}\. Crucially, the non\-linear cross\-attention fusion method achieves state\-of\-the\-art performance, demonstrating that neural language decoding benefits from simulating the collaborative modulation between contextual information and core lexical attributes rather than depending on isolated individual features, while also offering a viable non\-invasive brain\-to\-text decoding method\.

## 1Introduction

### 1\.1The Cross\-Modal Alignment Challenge in Semantic Reconstruction

Semantic reconstruction aims to translate neural signals into continuous natural text\[[2](https://arxiv.org/html/2607.12071#bib.bib1),[40](https://arxiv.org/html/2607.12071#bib.bib25)\]\. While speech decoders trained on invasive neural signals\[[4](https://arxiv.org/html/2607.12071#bib.bib60),[29](https://arxiv.org/html/2607.12071#bib.bib16),[48](https://arxiv.org/html/2607.12071#bib.bib61)\]leverage high\-SNR neural signals\[[13](https://arxiv.org/html/2607.12071#bib.bib4),[26](https://arxiv.org/html/2607.12071#bib.bib15)\]to yield robust results, their widespread usability is constrained by significant surgical risks\. Consequently, decoders based on safer, non\-invasive neural signals like magnetoencephalography \(MEG\) and electroencephalography \(EEG\) have emerged as scalable alternatives\. Prior efforts have explored the mapping of these non\-invasive neural recordings onto discrete semantic elements\[[9](https://arxiv.org/html/2607.12071#bib.bib62)\]or the benchmarking of representational differences\[[49](https://arxiv.org/html/2607.12071#bib.bib63)\]\. Concurrently, continuous language reconstruction has been demonstrated via fMRI voxels\[[40](https://arxiv.org/html/2607.12071#bib.bib25)\]and investigated using MEG\-based semantic decoders\[[45](https://arxiv.org/html/2607.12071#bib.bib28),[44](https://arxiv.org/html/2607.12071#bib.bib29)\], while the EEG modality has been introduced for auditory attention tracking, spectrogram reconstruction, and acoustic speech reconstruction\[[51](https://arxiv.org/html/2607.12071#bib.bib31),[52](https://arxiv.org/html/2607.12071#bib.bib32),[10](https://arxiv.org/html/2607.12071#bib.bib37)\], but using EEG for continuous text reconstruction remains extremely challenging\.

Despite these advances, non\-invasive semantic reconstruction suffers from a profound bottleneck: the representational mismatch between semantic feature spaces and neural coding patterns, which impedes accurate cross\-modal alignment\. Traditional frameworks mitigate this by regressing a single semantic feature, relying either on static lexical representations or contextualized representations in isolation\. Forcing a decoder to regress these isolated targets creates a severe representation gap due to inevitable information loss\. To facilitate a systematic analysis, we formalize these targets into a dual\-component configuration encompassing static linguistic attributes and dynamic contextual information\. From the static perspective, mainstream embedding models map individual words to fixed vector spaces based on text co\-occurrence statistics\[[30](https://arxiv.org/html/2607.12071#bib.bib17),[31](https://arxiv.org/html/2607.12071#bib.bib8),[34](https://arxiv.org/html/2607.12071#bib.bib18)\]\. While these static features capture stable conceptual coordinates, they represent words in complete isolation and omit essential contextual information\[[49](https://arxiv.org/html/2607.12071#bib.bib63)\], leading to semantic ambiguity over continuous text\. Conversely, the dynamic contextual component extracts representations from pre\-trained language models\[[12](https://arxiv.org/html/2607.12071#bib.bib3),[37](https://arxiv.org/html/2607.12071#bib.bib21)\]to track long\-range contextual sequences\[[38](https://arxiv.org/html/2607.12071#bib.bib23),[5](https://arxiv.org/html/2607.12071#bib.bib2),[46](https://arxiv.org/html/2607.12071#bib.bib30)\]\. Yet, relying solely on contextualized representations also introduces information loss, as these history\-dependent embeddings often blur sharp local word boundaries and dilute the stable baseline meaning of individual words, allowing high\-noise neural recordings to easily perturb discrete word retrieval\.

This single\-feature formulation directly deviates from the human brain’s language processing mechanisms\. Neurobiological evidence suggests that the human brain avoids this information loss by processing continuous language via two synergistic pathways: a lexical memory retrieval pathway that retrieves stable word attributes from distributed temporal networks\[[24](https://arxiv.org/html/2607.12071#bib.bib14)\], and a contextual unification pathway that dynamically integrates these units within fronto\-temporal networks\[[16](https://arxiv.org/html/2607.12071#bib.bib7),[17](https://arxiv.org/html/2607.12071#bib.bib9),[18](https://arxiv.org/html/2607.12071#bib.bib10)\]\. During continuous speech comprehension, the human brain simultaneously tracks stable lexical properties and contextual information to resolve semantic ambiguity\[[39](https://arxiv.org/html/2607.12071#bib.bib24),[24](https://arxiv.org/html/2607.12071#bib.bib14),[19](https://arxiv.org/html/2607.12071#bib.bib12),[35](https://arxiv.org/html/2607.12071#bib.bib20),[41](https://arxiv.org/html/2607.12071#bib.bib26),[14](https://arxiv.org/html/2607.12071#bib.bib6)\], interacting closely within the classic N400 time window\[[22](https://arxiv.org/html/2607.12071#bib.bib13)\]\.

Consequently, relying on a single isolated feature track is insufficient to capture the collaborative functions of human language networks\. To better match the biological language processing mechanisms, a robust semantic reconstruction framework requires a multi\-feature fusion approach\. By forcing static lexical representations and contextualized representations to modulate each other to synthesize a unified target semantic feature space, the framework can prevent information loss and facilitate accurate cross\-modal mapping\.

### 1\.2Proposed Framework and Core Contributions

Crucially, the engineering objective of designing a unified semantic feature space is to improve the computational feasibility of neural decoding, rather than mechanically replicating biological brain structures\. Since non\-invasive neural signals \(such as MEG and EEG\) are highly non\-stationary and extremely noisy, directly mapping them onto unconstrained language spaces can easily cause severe overfitting\. This vulnerability is further exacerbated by the time\-series autocorrelations inherent in non\-invasive neural approaches\[[50](https://arxiv.org/html/2607.12071#bib.bib33)\]\. A multi\-feature semantic space provides essential regularization; by constraining the decoding target within this well\-defined representational space, we structurally narrow the hypothesis space\. This constraint enables the decoding model to stably converge onto meaningful semantic targets despite the inherent noise of the neural data\.

To implement this target semantic feature space, this study introduces a multi\-feature fusion framework for semantic reconstruction\. Departing from traditional single\-feature approaches, we systematically formulate and contrast two alternative fusion methods to synthesize a unified semantic feature space: a linear fusion baseline via Naive Concatenation and a non\-linear fusion driven by Multi\-Head Cross\-Attention\. Within this cross\-attention method, the static lexical representations and dynamic contextual information interactively attend to one another, simulating the complementary mechanisms of collaborative multi\-feature processing native to human language networks\.

Evaluated cross\-modally on a continuous auditory narrative MEG dataset using self\-supervised contrastive alignment, our framework yields three primary contributions:

- •Dual\-Feature Fusion Framework:We propose a hybrid semantic feature target framework that integrates both static lexical properties and contextual information, improving the overall performance of both semantic reconstruction and continuous text generation\.
- •Support for Neuroscientific Hypotheses:By benchmarking linear and non\-linear feature integration, our experiments reveal a robust performance hierarchy\(Cross\-Att\>Concat\>GPT\>W2V\)\(\\text\{Cross\-Att\}\>\\text\{Concat\}\>\\text\{GPT\}\>\\text\{W2V\}\)\. The quantitative supremacy of reciprocal cross\-attention over naive concatenation provides support aligning with neuroscientific hypotheses regarding multi\-feature cooperative processing during language comprehension\.
- •Contrastive Alignment Method:We implement a self\-supervised contrastive learning method to optimize the cross\-modal framework\. This method effectively mitigates the difficulties inherent in semantic reconstruction using low SNR non\-invasive neural signals\.

![Refer to caption](https://arxiv.org/html/2607.12071v1/NCMMSCFIG4-3.png)Figure 1:Global architecture of the proposed interactive Multi\-feature fusion framework for non\-invasive semantic reconstruction\.The method is structured as a two\-stage framework: Stage 1: Semantic Feature Reconstruction and Contrastive Learning\. Text of audio stimuli are processed via tokenization to feed LM1\(Word2vec\) that captures stable lexical attributes and LM2\(GPT\-Neo\) that tracks dynamic sequence context\. These representations are integrated by the Fusion model to get fused semantic featuresEfusedE\_\{\\text\{fused\}\}\. Concurrently, neural recordings are mapped via a Neural Encoder into predicted fusion embedding\. A batch\-wide pairwise similarity matrix is constructed, governed by an InfoNCE loss function to maximize diagonal matches while minimizing off\-diagonal distractors\. Stage 2: Text Generation\. Conditioned on the historical prefix and the fusion embeddings predicted from Stage 1, pre\-trained LM3\(mengzi\-GPT\-Neo\) expands path hypotheses to generate candidate words\. Each candidate word is sequentially transformed into its fusion embedding via Fusion model\. The framework decides the next word by computing likehood between these candidate fusion embeddings and the predicted fusion embedding to output fluent, natural language text\.

## 2Method

In this section, we present the comprehensive methodology of our interactive Multi\-feature fusion framework for non\-invasive semantic reconstruction\. The global architecture of the proposed framework is illustrated in Fig\.[1](https://arxiv.org/html/2607.12071#S1.F1)\. The subsequent subsections will elaborate on the specific implementation and operational details of each component within the proposed framework\.

### 2\.1Methodological Overview and Design Motivations

Before formalizing the specific matrix operations, we outline the structural hypotheses governing our framework\. To model the dual facets of human language processing \(static lexical attributes and contextual information\), we utilize two distinct semantic feature extractors to construct a hybrid target space\. Crucially, we evaluate two opposite integration engineering methods: \(1\)Linear Concatenation Fusion, which hypothesizes that static lexicon items and context sequence tracks are processed via isolated, passive parallel channels, and \(2\)Interactive Cross\-Attention Fusion, which introduces active, reciprocal non\-linear gating blocks to let the two features dynamically modulate each other’s latent geometries\. These multi\-tier targets are then mathematically aligned with high\-dimensional neural time series using self\-supervised contrastive learning to drive downstream text generation methods\.

### 2\.2Dataset

SEM4Lang\.The SEM4Lang dataset comprises 60\.7 hours of magnetoencephalography \(MEG\) recordings gathered from 12 native Chinese speakers during a continuous speech listening task\[[47](https://arxiv.org/html/2607.12071#bib.bib35)\]\. Throughout the experiment, participants listened to 60 distinct audio stories, each spanning 4 to 7 minutes and encompassing a diverse range of topics\. Neural activity was captured at a sampling rate of 1,000 Hz using a 306\-channel whole\-head MEG system \(Elekta\-Neuromag TRIUX, Helsinki, Finland\)\.

### 2\.3Semantic Feature Extraction and Preprocessing

Semantic Feature Extraction\.Sixty Chinese audios were utilized as continuous natural narrative speech stimuli\. For the dynamic feature stream, we implemented a pre\-trained Chinese GPT\-Neo base model \(mengzi\-GPT\-Neo\-base\[[23](https://arxiv.org/html/2607.12071#bib.bib34)\], with a maximum sequence lengthM​L=128ML=128and a batch sizeB​S=64BS=64\)\. The hidden states of the 9th Transformer layer were extracted word\-by\-word utilizing a symmetrical contextual look\-back window of 5 tokens \(GPT\_WORDS=5\\text\{GPT\\\_WORDS\}=5\)\. Given a tokenized sequencew1,w2,…,wnw\_\{1\},w\_\{2\},\\dots,w\_\{n\}, the dynamic contextual vector for theii\-th token is formalized as:

eiGPT=GPT\-Neo\-layer\-9​\(\[wi−5,…,wi\]\)​\[−1,:\]∈ℝ768e\_\{i^\{\\text\{GPT\}\}\}=\\text\{GPT\-Neo\-layer\-9\}\(\[w\_\{i\-5\},\\dots,w\_\{i\}\]\)\[\-1,:\]\\in\\mathbb\{R\}^\{768\}\(1\)
Subsequent to the dynamic method, for the static feature of theii\-th token, 300\-dimensional word\-level representationeiW2Ve\_\{i^\{\\text\{W2V\}\}\}is extracted using a Continuous Bag\-of\-Words \(CBOW\) Word2Vec framework trained over a Chinese text corpus using a context window of 10\[[31](https://arxiv.org/html/2607.12071#bib.bib8),[30](https://arxiv.org/html/2607.12071#bib.bib17)\]\. After that, based on the start and end times of each token, its dynamic and static features are tiled across the entire time interval it occupies, yielding the final representationsEGPT∈ℝT×768E\_\{\\text\{GPT\}\}\\in\\mathbb\{R\}^\{T\\times 768\}andEW2V∈ℝT×300E\_\{\\text\{W2V\}\}\\in\\mathbb\{R\}^\{T\\times 300\}, whereTTrepresents the total number of sampling points of the entire story\. BothEGPTE\_\{\\text\{GPT\}\}andEW2VE\_\{\\text\{W2V\}\}feature sequences were downsampled to 40 Hz using linear interpolation to achieve exact temporal alignment with the neural signals\.

#### 2\.3\.1MEG Signal Preprocessing\.

MEG recordings collected across 12 healthy native Chinese speakers were preprocessed following standard artifact\-rejection methods\. For core regression mapping, only the 204 planar gradiometers were retained due to their superior signal\-to\-noise ratio \(SNR\) for localized cortical activity\. Signals underwent spatial denoising \(tSSS\), independent component analysis \(ICA\)\[[11](https://arxiv.org/html/2607.12071#bib.bib43),[28](https://arxiv.org/html/2607.12071#bib.bib44)\], lowpass filtering \(4 Hz\), and were downsampled to 40 Hz\. The preprocessing was implemented with MNE\-Python\[[15](https://arxiv.org/html/2607.12071#bib.bib36)\]\.

### 2\.4Semantic Feature Fusion

The overall detailed computational workflows for both the linear joint\-feature baseline and the advanced non\-linear interactive network are schematically illustrated in Fig\.[2](https://arxiv.org/html/2607.12071#S2.F2)\.

![Refer to caption](https://arxiv.org/html/2607.12071v1/NCMMSCFIG4-2.png)Figure 2:The architectural topology of the proposed feature fusion methods\. The upper block details the Concatenation Fusion baseline, where static vectors \(\[T,300\]\) and dynamic embeddings \(\[T,768\]\) are directly channel\-concatenated and linearly projected\.The lower block illustrates the Cross\-Attention Fusion framework, where individual linear layers first project both streams before routing them into reciprocal cross\-attention blocks\.#### 2\.4\.1Concatenation Fusion\.

Our first proposed method represents a joint\-feature baseline\. It simply concatenates the static Word2Vec vectorEW2V∈ℝT×300E\_\{\\text\{W2V\}\}\\in\\mathbb\{R\}^\{T\\times 300\}and the dynamic GPT vectorEGPT∈ℝT×768E\_\{\\text\{GPT\}\}\\in\\mathbb\{R\}^\{T\\times 768\}directly along the channel dimension\. While this baseline provides the brain decoder with access to both sources of information, it naively assumes that the human brain processes vocabulary and context through isolated parallel tracks without any non\-linear cross\-modal interaction\. Furthermore, after feature fusion, we apply a linear projection layer to unify the concatenated dimensions to 768, matching the input requirements of the downstream decoder\.

#### 2\.4\.2Cross\-Attention Fusion

To simulate the highly non\-linear, interactive feature synthesis native to human language unification, we introduce an advanced segment\-level multi\-modal fusion network \(FusionModel\) driven by a Multi\-Head Cross\-Attention block\[[20](https://arxiv.org/html/2607.12071#bib.bib48),[43](https://arxiv.org/html/2607.12071#bib.bib47),[36](https://arxiv.org/html/2607.12071#bib.bib46),[27](https://arxiv.org/html/2607.12071#bib.bib45)\]\. The detailed topological flow is formalized as follows:

Projection and Reshaping\.The input vectorsEW2VE\_\{\\text\{W2V\}\}andEGPTE\_\{\\text\{GPT\}\}are passed through individual Layer Normalization layers and linear projection layers to be mapped onto the target hidden dimensionDoutD\_\{\\text\{out\}\}\. They are then reshaped into a token sequence format ofntokens×dtokenn\_\{\\text\{tokens\}\}\\times d\_\{\\text\{token\}\}\(wheredtoken=Dout/ntokensd\_\{\\text\{token\}\}=D\_\{\\text\{out\}\}/n\_\{\\text\{tokens\}\}andntokensn\_\{\\text\{tokens\}\}is a tunable hyperparameter\) to get the feature matricesHGPTH\_\{\\text\{GPT\}\}andHW2VH\_\{\\text\{W2V\}\}:

HGPT=Reshape​\(LayerNorm​\(Linear​\(EW2V\)\)\)∈ℝT×ntokens×dtokenH\_\{\\text\{GPT\}\}=\\text\{Reshape\}\(\\text\{LayerNorm\}\(\\text\{Linear\}\(E\_\{\\text\{W2V\}\}\)\)\)\\in\\mathbb\{R\}^\{T\\times n\_\{\\text\{tokens\}\}\\times d\_\{\\text\{token\}\}\}\(2\)HW2V=Reshape​\(LayerNorm​\(Linear​\(EGPT\)\)\)∈ℝT×ntokens×dtokenH\_\{\\text\{W2V\}\}=\\text\{Reshape\}\(\\text\{LayerNorm\}\(\\text\{Linear\}\(E\_\{\\text\{GPT\}\}\)\)\)\\in\\mathbb\{R\}^\{T\\times n\_\{\\text\{tokens\}\}\\times d\_\{\\text\{token\}\}\}\(3\)
Cross\-Attention Interaction\.The two feature matrices serve asymmetric roles as Queries, Keys, and Values to perform dynamic cross\-gating and reciprocal filtering:

AttnGPT=MultiheadAttn​\(HGPT,HW2V,HW2V\)\\text\{Attn\}\_\{\\text\{GPT\}\}=\\text\{MultiheadAttn\}\(H\_\{\\text\{GPT\}\},H\_\{\\text\{W2V\}\},H\_\{\\text\{W2V\}\}\)\(4\)AttnW2V=MultiheadAttn​\(HW2V,HGPT,HGPT\)\\text\{Attn\}\_\{\\text\{W2V\}\}=\\text\{MultiheadAttn\}\(H\_\{\\text\{W2V\}\},H\_\{\\text\{GPT\}\},H\_\{\\text\{GPT\}\}\)\(5\)
The asymmetric cross\-gating mechanism and step\-by\-step tensor routing inside the primary interactive cell are explicitly mapped out in Fig\.[3](https://arxiv.org/html/2607.12071#S2.F3)\.

Fusion Output\.The bidirectional attention representations capturing complementary multi\-level semantics are concatenated and passed through a Multi\-Layer Perceptron \(MLP\) block, followed by temporal restoration:

Efused=MLP​\(\[AttnGPT;AttnW2V\]\)∈ℝT×DoutE\_\{\\text\{fused\}\}=\\text\{MLP\}\(\[\\text\{Attn\}\_\{\\text\{GPT\}\};\\text\{Attn\}\_\{\\text\{W2V\}\}\]\)\\in\\mathbb\{R\}^\{T\\times D\_\{\\text\{out\}\}\}\(6\)
![Refer to caption](https://arxiv.org/html/2607.12071v1/NCMMSCFIG4-1.png)Figure 3:The detailed computational method inside the Multi\-Head Cross\-Attention Block 1\. Static lexical representations provide the structural query matrixQQvia projection weightWqW\_\{q\}, while dynamic contextual vectors project into the corresponding key matrixKKand value matrixVVthroughWkW\_\{k\}andWvW\_\{v\}\. The scaled dot\-product interaction betweenQQandKKyields the dense attention matrixA​t​t​nAttn\. Finally, the value matrixVVis multiplied by the attention matrixA​t​t​nAttnto yieldA​t​t​nW​2​VAttn\_\{W2V\}\.

### 2\.5MEG to Text

The objective of this task is to directly decode and synthesize regular text narratives from non\-invasive neural signals acquired during continuous story listening\. As schematically outlined in Fig\.[1](https://arxiv.org/html/2607.12071#S1.F1), this neural decoding method is formalized into a decoupled, two\-stage cascade framework\.

#### 2\.5\.1Semantic Feature Reconstruction\.

Our primary objective is to evaluate the model’s capability to reconstruct comprehensive, multi\-tier semantic language spaces directly from non\-invasive neural signals\. Rather than performing sound\-level or wave\-level acoustic matching, this task focuses entirely on semantic feature mapping and token\-level lexical retrieval via the encoder network \(BrainNetwork, implementingConvConcatNet\[[52](https://arxiv.org/html/2607.12071#bib.bib32)\],BrainMagic\[[10](https://arxiv.org/html/2607.12071#bib.bib37)\],AwaveNet\[[25](https://arxiv.org/html/2607.12071#bib.bib50),[42](https://arxiv.org/html/2607.12071#bib.bib51)\],VLAAI\[[1](https://arxiv.org/html/2607.12071#bib.bib49)\]\)\. The mathematical alignment method and the batch\-wide contrastive optimization matrix used to bind the neural response embeddings with the interactive language targets are systematically detailed in Fig\.[1](https://arxiv.org/html/2607.12071#S1.F1)\.

Dataset Partitioning\.To evaluate zero\-shot generalization\[[32](https://arxiv.org/html/2607.12071#bib.bib52)\], the 60 trials from the SEM4Lang dataset are randomly split into a training set of 40 trials, a validation set of 10 trials, and a test set of 10 trials\. Continuous data are sliced into uniform, non\-overlapping intervals with an epoch duration ofs​e​g​l​e​n=3​sseglen=3\\text\{s\}across all subject sessions\.

Self\-Supervised Contrastive Learning and Loss Function\.The alignment method binds the neural response embeddings with the interactive multi\-tier language targets\. To capture exact temporal synchronization across the time series, the network minimizes a batch\-wide InfoNCE contrastive loss\[[6](https://arxiv.org/html/2607.12071#bib.bib38),[7](https://arxiv.org/html/2607.12071#bib.bib39)\]based on a pairwise temporal Pearson correlation similarity matrix\. Within a mini\-batch of sizeBB, letE~i\\tilde\{E\}\_\{i\}represent the brain encoder’s prediction for theii\-th sample of the batch andEfused,jE\_\{\\text\{fused\},j\}denote the true fused target for thejj\-th sample of the batch\. The time\-step wise Pearson similarity coefficient\(i,j\)\(i,j\)over the continuous temporal durationTT\(T=s​e​g​l​e​n×40T=seglen\\times 40\) is formalized as:

s​i​m​\(i,j\)=∑t=1T\(E~i,t−E~¯i,t\)​\(Ej,tfused−E¯j,tfused\)\(Dout−1\)⋅σ​\(E~i,t\)⋅σ​\(Ej,tfused\)sim\(i,j\)=\\sum\_\{t=1\}^\{T\}\\frac\{\(\\tilde\{E\}\_\{i,t\}\-\\overline\{\\tilde\{E\}\}\_\{i,t\}\)\(E\_\{j,t\}^\{\\text\{fused\}\}\-\\overline\{E\}\_\{j,t\}^\{\\text\{fused\}\}\)\}\{\(D\_\{\\text\{out\}\}\-1\)\\cdot\\sigma\(\\tilde\{E\}\_\{i,t\}\)\\cdot\\sigma\(E\_\{j,t\}^\{\\text\{fused\}\}\)\}\(7\)whereE¯\\overline\{E\}andσ​\(⋅\)\\sigma\(\\cdot\)denote the mean and standard deviation of the feature vector along the channel dimension, respectively\. Using this pairwise similarity metric, the batch\-wide contrastive InfoNCE loss functionℒ\\mathcal\{L\}is constructed as:

ℒ=−1B​∑i=1Blog⁡exp⁡\(s​i​m​\(i,i\)\)∑j=1Bexp⁡\(s​i​m​\(i,j\)\)\\mathcal\{L\}=\-\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\log\\frac\{\\exp\(sim\(i,i\)\)\}\{\\sum\_\{j=1\}^\{B\}\\exp\(sim\(i,j\)\)\}\(8\)By maximizing the diagonal scoress​i​m​\(i,i\)sim\(i,i\)and compressing the off\-diagonal entriess​i​m​\(i,j\)sim\(i,j\), the loss forces the networks to map true brain\-text associations together while pushing away mismatched negative distractors within the same batch\.

#### 2\.5\.2Text Generation\.

This phase reconstructs continuous natural language word\-by\-word from neural signals\.

Conditioned on the reconstructed neural embedding tensorsE~\\tilde\{E\}from Stage 1, a heuristic beam search algorithm synthesizes text strings across discrete word timestampst1,t2,…,tMt\_\{1\},t\_\{2\},\\dots,t\_\{M\}over a deduplicated decoding vocabulary𝒱\\mathcal\{V\}\(3,5353,535tokens\), which merges high\-frequency Chinese characters \(≥2\\geq 2occurrences in the first 50 stories\) and themengzi\-GPT\-Neotokenizer\.

For each candidate tokenww, its virtual fused representation is obtained via:

E​\(w\)=fusion​\(eW2V​\(w\),hGPT​\(w\)\)E\(w\)=\\text\{fusion\}\(e\_\{\\text\{W2V\}\}\(w\),h\_\{\\text\{GPT\}\}\(w\)\)\(9\)wherehGPT​\(w\)h\_\{\\text\{GPT\}\}\(w\)is the hidden state vector from the 9th Transformer layer computed by appendingwwto the current path prefix inside theGeneration LMunder a contextual window ofGPT\_WORDS=5\\text\{GPT\\\_WORDS\}=5\. The static vectoreW2V​\(w\)e\_\{\\text\{W2V\}\}\(w\)is indexed via a word look\-up matrix𝒥:Σ→ℝ300\\mathcal\{J\}:\\Sigma\\rightarrow\\mathbb\{R\}^\{300\}\.

To balance efficiency and fluency, the framework deploys a brain\-guided heuristic beam search with a fixed beam widthB=100B=100and step expansion parameterE=5E=5\. The recursive iteration proceeds as follows:

1. 1\.Initialization:The search tree is seeded withEEhigh\-probability root nodes from the narrative starter pool in the tracking beam\.
2. 2\.Temporal Prior Extension:For each word indexk=2,3,…,Mk=2,3,\\dots,M, the preceding tokens of each surviving pathh∈ℬk−1h\\in\\mathcal\{B\}\_\{k\-1\}are fed into theGeneration LM\(bounded by a lower floor of 5 tokens and an upper threshold ofLM\_TIME=8\\text\{LM\\\_TIME\}=8seconds\) to output the conditional probabilityPLM​\(w\|hctx\)P\_\{\\text\{LM\}\}\(w\|h\_\{\\text\{ctx\}\}\)\. Top\-KKsampling \(t​o​p​\-​k=10top\\text\{\-\}k=10\) isolates a reduced candidate list𝒞h\\mathcal\{C\}\_\{h\}while filtering repetitive stop\-words\.
3. 3\.Physiological Correlation Scoring:Each candidatew∈𝒞hw\\in\\mathcal\{C\}\_\{h\}is converted intoe​\(w\)e\(w\)and cross\-matched against the reconstructed neural response segmentE~​\[tk,tk\+1\]\\tilde\{E\}\[t\_\{k\},t\_\{k\+1\}\]from Stage 1\. The brain\-responsive matching score is derived via cross\-time average Pearson correlation coefficients: sbrain​\(w\)=1tk\+1−tk​∑t=tktk\+1Pearson​\(E​\(w\),E~​\[t\]\)s\_\{\\text\{brain\}\}\(w\)=\\frac\{1\}\{t\_\{k\+1\}\-t\_\{k\}\}\\sum\_\{t=t\_\{k\}\}^\{t\_\{k\+1\}\}\\text\{Pearson\}\(E\(w\),\\tilde\{E\}\[t\]\)\(10\)
4. 4\.Adaptive Pruning:Paths are allocated an expansion budgetnh=\|E⋅rank​\(b\)\|ℬk−1\|\|n\_\{h\}=\|E\\cdot\\frac\{\\text\{rank\}\(b\)\}\{\|\\mathcal\{B\}\_\{k\-1\}\|\}\|based on their accumulated LM log\-probabilities\. All expanded child hypotheses are pooled and sorted globally in descending order ofsbrain​\(w\)s\_\{\\text\{brain\}\}\(w\), preserving only the topBBpaths: ℬk=Top\-​Bw∈∪h𝒞h​\(sbrain​\(w\)\)\\mathcal\{B\}\_\{k\}=\\text\{Top\-\}B\_\{w\\in\\cup\_\{h\}\\mathcal\{C\}\_\{h\}\}\(s\_\{\\text\{brain\}\}\(w\)\)\(11\)

### 2\.6Evaluation

#### 2\.6\.1Semantic Feature Reconstruction\.

The reconstruction fidelity is quantitatively evaluated using a Top\-KKRank Accuracy metric\[[21](https://arxiv.org/html/2607.12071#bib.bib53)\]\. All sub\-segments from the testing set are cross\-matched against the complete reference pool based on their temporal Pearson similarity scores\. The candidates are sorted in descending order according to their similarity coefficients\. A retrieval instance is classified as successful if the matched ground\-truth semantic vector falls within the topKKpositions of the sorted candidate array, whereKKis 1 or 10\.

#### 2\.6\.2Text Generation\.

The final text generation outputs are cross\-matched against the ground\-truth text transcripts at a character level across local temporal sliding windows \(WINDOW=20\\text\{WINDOW\}=20seconds\) to evaluate structural and semantic decoding stability over long narrative durations\. The translation and language similarity benchmarks comprise:BLEU\-1\(measures token precision via unigram exact matches\)\[[33](https://arxiv.org/html/2607.12071#bib.bib55)\],METEOR\(evaluates structural generation alignment by calculating comprehensive precision and recall scores\)\[[3](https://arxiv.org/html/2607.12071#bib.bib54)\], andBERTScore\(computes non\-linear contextual semantic similarity leveraging layer\-8 hidden state recall values from a pre\-trainedbert\-base\-chinesemodel\)\[[53](https://arxiv.org/html/2607.12071#bib.bib56)\]\.

To ensure that the text generation performance is heavily driven by neural grounding rather than language model hallucinations, we establish aNull Baseline\[[8](https://arxiv.org/html/2607.12071#bib.bib59)\]\. At each forward token iteration, the real cortical correlation scoresbrain​\(w\)s\_\{\\text\{brain\}\}\(w\)is blocked and replaced with a uniform random variable∼𝒰​\(0,1\)\\sim\\mathcal\{U\}\(0,1\), keeping all other search and pruning parameters identical\[[50](https://arxiv.org/html/2607.12071#bib.bib33)\]\. Comparing the authentic fused performance against this opportunity baseline verifies the statistical significance of our non\-invasive neuro\-decoding framework\.

## 3Results

### 3\.1Semantic Feature Reconstruction

We baseline four neural encoders \(ConvConcatNet\[[52](https://arxiv.org/html/2607.12071#bib.bib32)\],BrainMagicNet\[[44](https://arxiv.org/html/2607.12071#bib.bib29)\],AWaveNet\[[25](https://arxiv.org/html/2607.12071#bib.bib50)\], andVLAAI\[[1](https://arxiv.org/html/2607.12071#bib.bib49)\]\) across four distinct linguistic and contextual configurations under a zero\-shot semantic retrieval task\(Fig\.[4](https://arxiv.org/html/2607.12071#S3.F4)\)\.

Performance Hierarchy and Complementary Integration\.The empirical retrieval metrics reveal a robust and consistent performance hierarchy across all neural decoders and temporal segments, adhering to the inequality chain:Cross\-Att\>Concat\>GPT\>W2V\\text\{Cross\-Att\}\>\\text\{Concat\}\>\\text\{GPT\}\>\\text\{W2V\}\(Fig\.[4](https://arxiv.org/html/2607.12071#S3.F4)\)\. Specifically, the non\-linear fusion method \(Cross\-Att\) significantly outperforms the linear joint\-feature baseline \(Concat\) across multiple time windows \(p∗<0\.05\{\}^\{\*\}\\,p<0\.05,p∗∗<0\.01\{\}^\{\*\*\}\\,p<0\.01,p∗⁣∗∗<0\.001\{\}^\{\*\*\*\}\\,p<0\.001\), while both unified approaches surpass the standalone single\-component spaces\. This substantial gain demonstrates that complements between static linguistic attributes and dynamic contextual information can effectively elevate decoding capacity\.

Compared to the passive linear concatenation baseline, the non\-linear cross\-attention model achieves significantly tighter lexical attribute clustering and higher cross\-modal alignment accuracy\. This performance gap empirically demonstrates that simulating the interactive integration and collaborative modulation between static lexical properties and contextual profiles allows the decoding framework to yield superior representational alignment with endogenous neural dynamics\.

![Refer to caption](https://arxiv.org/html/2607.12071v1/result.png)Figure 4:Quantitative evaluation of semantic reconstruction performance measured by Top\-1 \(left column\) and Top\-10 \(right column\) retrieval accuracies across varying temporal window sizes \(3 s, 5 s, and 10 s\)\. Four specialized neural decoders \(ConvConcat,BrainMagic,AWaveNet, andVLAAI\) are benchmarked across four linguistic and contextual configurations: static word embeddings \(W2V\), standalone contextual representations \(GPT\), linear joint features \(Concat\), and the proposed non\-linear fusion method \(Cross\-Att\)\. Bar heights represent the mean decoding accuracy, and error bars denote the standard error of the mean \(SEM\) computed across subject sessions \(n=12n=12\)\. Horizontal lines indicate statistical significance derived from paired one\-tailedtt\-tests \(p∗<0\.05\{\}^\{\*\}\\,p<0\.05,p∗∗<0\.01\{\}^\{\*\*\}\\,p<0\.01,p∗⁣∗∗<0\.001\{\}^\{\*\*\*\}\\,p<0\.001\)\.Pronounced Temporal Slicing Effect\.The decoding metrics reveal a strong positive correlation between temporal context duration \(seglen\) and retrieval robustness\. While retrieval in short windows \(s​e​g​l​e​n=3​sseglen=3\\,\\text\{s\}\) remains highly challenging due to sparse contextual information and the inherently low SNR of non\-invasive recordings, extending the continuous epoch length to10​s10\\,\\text\{s\}triggers a pronounced surge in Top\-10 accuracy \(Fig\.[4](https://arxiv.org/html/2607.12071#S3.F4)\)\. This indicates that aggregating longer neural time series effectively smooths out high\-frequency biological noise, elevates the signal\-to\-noise ratio, and provides stable global semantic boundaries to alleviate cross\-modal alignment difficulties\[[50](https://arxiv.org/html/2607.12071#bib.bib33)\]\.

Neural Architecture Comparison\.Among the specialized neural encoders,ConvConcatNetandBrainMagicNetemerge as the top\-performing encoders\. Specifically,BrainMagicNetpaired with theCross\-Attconfiguration achieves prominent retrieval ceilings in the5​s5\\,\\text\{s\}and10​s10\\,\\text\{s\}tasks, whileConvConcatNetexhibits remarkable stability across all feature configurations\. This validates that multi\-scale temporal convolutions excel at capturing stable, localized neural states from non\-stationary neuroimaging dynamics\[[52](https://arxiv.org/html/2607.12071#bib.bib32)\]\.

### 3\.2Text Generation

To evaluate end\-to\-end decoding proficiency, the top two neural encoders \(BrainMagicNetandConvConcatNet\) were interfaced with the downstream text generation stage, quantified via character\-level metrics: BLEU\-1\[[33](https://arxiv.org/html/2607.12071#bib.bib55)\], METEOR\[[3](https://arxiv.org/html/2607.12071#bib.bib54)\], and BERTScore\[[53](https://arxiv.org/html/2607.12071#bib.bib56)\]\(Table[1](https://arxiv.org/html/2607.12071#S3.T1)\)\.

Table 1:Quantitative text generation results across different neural encoders and semantic feature configurations\. Bold values indicate the top\-performing setup \(performance ceiling\) within each metrics category\.Configuration Hierarchy and Architectural Synthesis\.The superiority of our advanced brain\-to\-text method is directly grounded in these empirical generation evaluations\. As detailed in Table[1](https://arxiv.org/html/2607.12071#S3.T1), theCross\-Attframework consistently yields the highest generation quality, achieving a BLEU\-1 score of15\.58±1\.1615\.58\\pm 1\.16and a METEOR score of9\.23±0\.899\.23\\pm 0\.89when paired withConvConcatNet, outperforming the Null Baseline and standalone single\-component configurations by a wide, statistically significant margin\.

ConvConcatNetdemonstrates a pronounced advantage overBrainMagicNetacross all configurations\. This performance gap suggests that multi\-scale temporal convolutional frameworks excel at capturing stable, granular local neural coding patterns required for precise sequence generation\. Furthermore, both decoders significantly surpass the Chance Level \(Null\) baseline, demonstrating that the downstream text generation performance is genuinely driven by actual cortical alignment and neural grounding, rather than by deep language model hallucinations\.

## 4Conclusion

### 4\.1Summary and Implications

This study introduced a multi\-feature fusion framework to address the representational mismatch between computational semantic features and multi\-channel non\-invasive neural recordings\. By transitioning from standalone, single\-feature decoding targets to synthesized representation spaces, we formalized and benchmarked linear Naive Concatenation against a non\-linear network governed by Multi\-Head Cross\-Attention\.

Our semantic reconstruction and text generation experiments revealed a consistent performance hierarchy\(Cross\-Att\>Concat\>GPT\>W2V\)\(\\text\{Cross\-Att\}\>\\text\{Concat\}\>\\text\{GPT\}\>\\text\{W2V\}\)across multiple architectures, backed by statistical validation\. The empirical success of the cross\-attention method demonstrates that non\-invasive language decoding benefits from simulating the interactive integration and collaborative modulation between static linguistic attributes and dynamic contextual information, establishing a target space that improves cross\-modal alignment accuracy\. This multi\-feature synthesis establishes a target representation space that is computationally isomorphic to the neural coding patterns of the human language network\. By constraining the decoding model within this cognitively grounded framework, we reduce the search space against the non\-stationary noise floor inherent in non\-invasive neuroimaging, thereby providing a robust method for continuous brain\-to\-text reconstruction\.

### 4\.2Future Work

To transition this framework toward practical BCI deployment, future research will pursue two primary directions:

- •Cross\-Subject Generalization:To mitigate prominent inter\-subject variability, we aim to incorporate adversarial domain adaptation layers to align latent neural signals across subjects, thereby minimizing the calibration overhead for novel users\.
- •Open\-Source Benchmarking:To foster reproducibility and collaborative development, we will release our multi\-encoder encoders, preprocessing approaches, and text generation modules as a standardized, open\-source Python library to provide a unified evaluation platform for the BCI community\.

## References

- \[1\]B\. Accou, J\. Vanthornhout, and V\. hamme\(2023\)Decoding of the speech envelope from EEG using the VLAAI deep neural network\.Scientific Reports13\(1\),pp\. 812\.Cited by:[§2\.5\.1](https://arxiv.org/html/2607.12071#S2.SS5.SSS1.p1.1.5),[§3\.1](https://arxiv.org/html/2607.12071#S3.SS1.p1.1)\.
- \[2\]G\. K\. Anumanchipalli, J\. Chartier, and E\. F\. Chang\(2019\)Speech synthesis from neural decoding of spoken sentences\.Nature568\(7753\),pp\. 493–498\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[3\]S\. Banerjee and A\. Lavie\(2005\)METEOR: an automatic metric for MT evaluation with improved correlation with human judgments\.InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization,pp\. 65–72\.Cited by:[§2\.6\.2](https://arxiv.org/html/2607.12071#S2.SS6.SSS2.p1.1),[§3\.2](https://arxiv.org/html/2607.12071#S3.SS2.p1.1)\.
- \[4\]S\. Card, M\. Wairagkar, and C\. Iacobacci\(2024\)An accurate and rapidly calibrating speech neuroprosthesis\.New England Journal of Medicine391\(7\),pp\. 609–618\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[5\]C\. Caucheteux and J\. King\(2022\)Brains and algorithms partially converge in natural language processing\.Communications Biology5\(1\),pp\. 134\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p2.1)\.
- \[6\]T\. Chen, S\. Kornblith, M\. Norouzi, and G\. Hinton\(2020\)A simple framework for contrastive learning of visual representations\.InProceedings of the 37th International Conference on Machine Learning,pp\. 1597–1607\.Cited by:[§2\.5\.1](https://arxiv.org/html/2607.12071#S2.SS5.SSS1.p3.8)\.
- \[7\]X\. Chen and K\. He\(2021\)Exploring simple siamese representation learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 15750–15758\.Cited by:[§2\.5\.1](https://arxiv.org/html/2607.12071#S2.SS5.SSS1.p3.8)\.
- \[8\]E\. Combrisson and K\. Jerbi\(2015\)Exceeding chance level by chance: the caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy\.Journal of Neuroscience Methods250,pp\. 126–136\.Cited by:[§2\.6\.2](https://arxiv.org/html/2607.12071#S2.SS6.SSS2.p2.2.1)\.
- \[9\]S\. d’Ascoli, C\. Bel, and J\. Rapin\(2025\)Towards decoding individual words from non\-invasive brain recordings\.Nature Communications16\(1\),pp\. 10521\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[10\]A\. Défossez, C\. Caucheteux, J\. Rapin, O\. Kabeli, and J\. King\(2023\)Decoding speech perception from non\-invasive brain recordings\.Nature Machine Intelligence5\(10\),pp\. 1097–1107\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1),[§2\.5\.1](https://arxiv.org/html/2607.12071#S2.SS5.SSS1.p1.1.3)\.
- \[11\]A\. Delorme and S\. Makeig\(2004\)EEGLAB: an open source toolbox for analysis of single\-trial EEG dynamics including independent component analysis\.InJournal of Neuroscience Methods,Vol\.134,pp\. 9–21\.Cited by:[§2\.3\.1](https://arxiv.org/html/2607.12071#S2.SS3.SSS1.p1.1)\.
- \[12\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4171–4186\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p2.1)\.
- \[13\]S\. Duraivel, S\. Rahimpour, C\. Chiang,et al\.\(2023\)High\-resolution neural recordings improve the accuracy of speech decoding\.Nature Communications14\(1\),pp\. 6938\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[14\]A\. Goldstein, Z\. Zada, and E\. Buchnik\(2022\)Shared computational principles for language processing in humans and deep language models\.Nature Neuroscience25\(3\),pp\. 369–380\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[15\]A\. Gramfort, M\. Luessi, E\. Larson, D\. A\. Engemann, D\. Strohmeier, C\. Brodbeck, L\. Parkkonen, and M\. S\. Hämäläinen\(2014\)MNE software for processing MEG and EEG data\.NeuroImage86,pp\. 446–460\.Cited by:[§2\.3\.1](https://arxiv.org/html/2607.12071#S2.SS3.SSS1.p1.1)\.
- \[16\]P\. Hagoort\(2005\)On Broca, brain, and binding: a new framework\.Trends in Cognitive Sciences9\(9\),pp\. 416–423\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[17\]P\. Hagoort\(2013\)MUC \(Memory, Unification, Control\) and beyond\.Frontiers in Psychology4,pp\. 416\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[18\]P\. Hagoort\(2019\)The neurobiology of language beyond single\-word processing\.Science366\(6461\),pp\. 55–58\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[19\]S\. Jain and A\. G\. Huth\(2018\)Incorporating context into language encoding models for fMRI\.InAdvances in Neural Information Processing Systems,Vol\.31,pp\. 6628–6629\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[20\]K\. Jiang, T\. Zhang, and X\. Liu\(2022\)Cross\-modality transformer for visible\-infrared person re\-identification\.InProceedings of the 16th European Conference on Computer Vision,pp\. 480–496\.Cited by:[§2\.4\.2](https://arxiv.org/html/2607.12071#S2.SS4.SSS2.p1.1)\.
- \[21\]T\. N\. Kay, T\. Naselaris, R\. J\. Prenger, and J\. L\. Gallant\(2008\)Identifying natural images from human brain activity\.Nature452\(7185\),pp\. 352–355\.Cited by:[§2\.6\.1](https://arxiv.org/html/2607.12071#S2.SS6.SSS1.p1.3)\.
- \[22\]M\. Kutas and K\. D\. Federmeier\(2011\)Thirty years and counting: finding meaning in the N400 component of the event\-related brain potential \(ERP\)\.Annual Review of Psychology62,pp\. 621–647\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[23\]Langboat Technology\(2021\)Mengzi\-GPT\-Neo\-base: A pre\-trained Chinese language model\.Note:[https://huggingface\.co/Langboat/mengzi\-gpt\-neo\-base](https://huggingface.co/Langboat/mengzi-gpt-neo-base)Cited by:[§2\.3](https://arxiv.org/html/2607.12071#S2.SS3.p1.5.2)\.
- \[24\]E\. F\. Lau, C\. Phillips, and D\. Poeppel\(2008\)A cortical network for semantics: \(de\)constructing the N400\.Nature Reviews Neuroscience9\(12\),pp\. 920–933\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[25\]H\. Li, Y\. Fang, X\. Zhang, F\. Chen, and G\. Gao\(2024\)Cross\-Attention\-Guided WaveNet for EEG\-to\-MEL spectrogram reconstruction\.InProceedings of the 25th Annual Conference of the International Speech Communication Association \(Interspeech 2024\),pp\. 2620–2624\.Cited by:[§2\.5\.1](https://arxiv.org/html/2607.12071#S2.SS5.SSS1.p1.1.4),[§3\.1](https://arxiv.org/html/2607.12071#S3.SS1.p1.1)\.
- \[26\]J\. Li, C\. Guo, C\. Zhang, E\. F\. Chang, and Y\. Li\(2026\)High\-fidelity neural speech reconstruction through an efficient acoustic\-linguistic dual\-pathway framework\.eLife14,pp\. RP109400\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[27\]J\. Lu, D\. Batra, D\. Parikh, and S\. Lee\(2019\)ViLBERT: pretraining task\-agnostic visiolinguistic representations for vision\-and\-language tasks\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 13–23\.Cited by:[§2\.4\.2](https://arxiv.org/html/2607.12071#S2.SS4.SSS2.p1.1)\.
- \[28\]S\. Makeig, A\. J\. Bell, T\. Jung, and T\. J\. Sejnowski\(1995\)Independent component analysis of electroencephalographic data\.InAdvances in Neural Information Processing Systems,Vol\.8,pp\. 145–151\.Cited by:[§2\.3\.1](https://arxiv.org/html/2607.12071#S2.SS3.SSS1.p1.1)\.
- \[29\]S\. L\. Metzger, M\. K\. Littlejohn, D\. B\. Silva, P\. Carattini, C\. M\. Coogan, J\. R\. Liu, and E\. F\. Chang\(2023\)A high\-performance neuroprosthesis for speech decoding and avatar control\.Nature620\(7976\),pp\. 1037–1046\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[30\]T\. Mikolov, K\. Chen, G\. Corrado, and J\. Dean\(2013\)Efficient estimation of word representations in vector space\.arXiv preprint arXiv:1301\.3781\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p2.1),[§2\.3](https://arxiv.org/html/2607.12071#S2.SS3.p2.7)\.
- \[31\]T\. Mikolov, I\. Sutskever, K\. Chen, G\. S\. Corrado, and J\. Dean\(2013\)Distributed representations of words and phrases and their compositionality\.Advances in neural information processing systems26\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p2.1),[§2\.3](https://arxiv.org/html/2607.12071#S2.SS3.p2.7)\.
- \[32\]M\. Palatucci, D\. Pomerleau, G\. E\. Hinton, and T\. M\. Mitchell\(2009\)Zero\-shot learning with semantic output codes\.Advances in Neural Information Processing Systems22,pp\. 1410–1418\.Cited by:[§2\.5\.1](https://arxiv.org/html/2607.12071#S2.SS5.SSS1.p2.1)\.
- \[33\]K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu\(2002\)BLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§2\.6\.2](https://arxiv.org/html/2607.12071#S2.SS6.SSS2.p1.1),[§3\.2](https://arxiv.org/html/2607.12071#S3.SS2.p1.1)\.
- \[34\]J\. Pennington, R\. Socher, and C\. D\. Manning\(2014\)GloVe: global vectors for word representation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1532–1543\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p2.1)\.
- \[35\]L\. Pylkkänen\(2019\)The neural basis of combinatory syntax and semantics\.Science363\(6425\),pp\. 62–66\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[36\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever\(2021\)Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,pp\. 8748–8763\.Cited by:[§2\.4\.2](https://arxiv.org/html/2607.12071#S2.SS4.SSS2.p1.1)\.
- \[37\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p2.1)\.
- \[38\]M\. Schrimpf, A\. Blank, and G\. Tuckute\(2021\)The neural architecture of language: integrative modeling converges on predictive processing\.Proceedings of the National Academy of Sciences118\(45\),pp\. e2105646118\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p2.1)\.
- \[39\]S\. C\. Sereno, C\. R\. Brewer, and P\. J\. O’Donnell\(2003\)Context effects in word recognition: evidence for early interactive processing\.Psychological Science14\(4\),pp\. 328–333\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[40\]J\. Tang, A\. LeBel, S\. Jain, and A\. G\. Huth\(2023\)Semantic reconstruction of continuous language from non\-invasive brain recordings\.Nature Neuroscience26\(5\),pp\. 858–866\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[41\]M\. Toneva, T\. M\. Mitchell, and L\. Wehbe\(2022\)Combining computational controls with natural text reveals aspects of meaning composition\.Nature Computational Science2\(11\),pp\. 745–757\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p3.1)\.
- \[42\]B\. Van Dyck, L\. Yang, and M\. M\. Van Hulle\(2023\)Decoding auditory EEG responses using an adapted Wavenet\.InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP 2024\),pp\. 1–2\.Cited by:[§2\.5\.1](https://arxiv.org/html/2607.12071#S2.SS5.SSS1.p1.1.4)\.
- \[43\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 5998–6008\.Cited by:[§2\.4\.2](https://arxiv.org/html/2607.12071#S2.SS4.SSS2.p1.1)\.
- \[44\]B\. Wang, X\. Xu, B\. Xiao, L\. Zheng, X\. Wu, H\. Cheng, and J\. Chen\(2026\)Hierarchical decoding of perceived speech from non\-invasive brain recordings\.IEEE Transactions on Neural Systems and Rehabilitation Engineering34,pp\. 2349–2360\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.12071#S3.SS1.p1.1)\.
- \[45\]B\. Wang, X\. Xu, L\. Zhang, B\. Xiao, X\. Wu, and J\. Chen\(2024\)Semantic reconstruction of continuous language from MEG signals\.InProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP 2024\),pp\. 2190–2194\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[46\]B\. Wang, X\. Xu, Z\. Zhang, H\. Zhu, Y\. Yan, X\. Wu, and J\. Chen\(2024\)Self\-supervised speech representation and contextual text embedding for match\-mismatch classification with EEG recording\.InProceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops \(ICASSPW 2024\),pp\. 111–112\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p2.1)\.
- \[47\]S\. Wang, X\. Zhang, J\. Zhang, and C\. Zong\(2022\)A synchronized multimodal neuroimaging dataset for studying brain language processing\.Scientific Data9\(1\),pp\. 590\.Cited by:[§2\.2](https://arxiv.org/html/2607.12071#S2.SS2.p1.1)\.
- \[48\]R\. Willett, M\. Kunz, and C\. Fan\(2023\)A high\-performance speech neuroprosthesis\.Nature620\(7976\),pp\. 1031–1036\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[49\]B\. Xiao, B\. Wang, X\. Chen, X\. z\. Xu, X\. Wu, and J\. Chen\(2024\)Comparing human\-labeled and LLM\-generated semantic features via cortical neural representation\.InProceedings of the 14th International Symposium on Chinese Spoken Language Processing \(ISCSLP 2024\),pp\. 666–670\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1),[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p2.1)\.
- \[50\]X\. Xu, B\. Wang, B\. Xiao, N\. Yadong, Y\. Wang, X\. Wu, H\. Cheng, and J\. Chen\(2026\)The impacts of temporal autocorrelations on EEG decoding\.Biomedical Signal Processing and Control113,pp\. 108783\.Cited by:[§1\.2](https://arxiv.org/html/2607.12071#S1.SS2.p1.1),[§2\.6\.2](https://arxiv.org/html/2607.12071#S2.SS6.SSS2.p2.2),[§3\.1](https://arxiv.org/html/2607.12071#S3.SS1.p4.2)\.
- \[51\]X\. Xu, B\. Wang, Y\. Yan, X\. Wu, and J\. Chen\(2024\)A DenseNet\-based method for decoding auditory spatial attention with EEG\.InProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP 2024\),pp\. 1946–1950\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1)\.
- \[52\]X\. Xu, B\. Wang, Y\. Yan, H\. Zhu, Z\. Zhang, X\. w\. Wu, and J\. Chen\(2024\)ConvConcatNet: a deep convolutional neural network to reconstruct mel spectrogram from the EEG\.InProceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops \(ICASSPW 2024\),pp\. 113–114\.Cited by:[§1\.1](https://arxiv.org/html/2607.12071#S1.SS1.p1.1),[§2\.5\.1](https://arxiv.org/html/2607.12071#S2.SS5.SSS1.p1.1.2),[§3\.1](https://arxiv.org/html/2607.12071#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.12071#S3.SS1.p5.2)\.
- \[53\]T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi\(2019\)BERTScore: evaluating text generation with BERT\.InInternational Conference on Learning Representations,Cited by:[§2\.6\.2](https://arxiv.org/html/2607.12071#S2.SS6.SSS2.p1.1),[§3\.2](https://arxiv.org/html/2607.12071#S3.SS2.p1.1)\.