Sleep2Word:通过时长感知词元化实现外周信号到脑电信号的转换

arXiv cs.CL 论文

摘要

Sleep2Word 提出了一种框架,利用具有时长感知能力的生理词元,将外周信号(呼吸、ECG、PPG)转换为整夜 EEG 频谱图。在涵盖 2 万余例多导睡眠监测(PSG)病例的 9 个数据集上,该框架在 EEG 重建、睡眠分期、觉醒检测和生存预测任务中的表现均优于基线方法。

arXiv:2611.21643v1 Announce Type: new Abstract: Estimating full-night electroencephalography (EEG) spectrograms from peripheral measurements could extend EEG-informed sleep analysis to recordings acquired without EEG. This task requires compact representations of signals with different temporal structures. We introduce Sleep2Word, a framework for translating respiration, electrocardiography (ECG), or photoplethysmography (PPG) into EEG spectrograms through duration-aware physiological words. Building on context-dependent chunking, Sleep2Word groups physiological embeddings into variable-duration chunks and jointly quantizes their content and duration into modality-specific wordbooks. A block-autoregressive translator then predicts target EEG word identities and occurrence-specific duration refinements within fixed-duration blocks. This design accommodates independent source and target wordbooks, limits temporal drift, and shortens the autoregressive decoding path. Trained on over 20K PSG cases across 9 datasets, Sleep2Word consistently outperforms existing spectrogram generation baselines in EEG spectrogram reconstruction, sleep staging, arousal detection, and survival prediction. Compressing full night sequences by up to 69.7%, the learned words yield a favorable accuracy-compute trade-off. Sleep event analysis further shows that the learned words capture physiologically plausible associations across modalities, suggesting their potential for interpretable cross-modal analysis.
查看原文
查看缓存全文

缓存时间: 2026/11/19 02:58

# Sleep2Word: Peripheral-to-EEG Translation via Duration-Aware Wordization
Source: [https://arxiv.org/html/2611.21643](https://arxiv.org/html/2611.21643)
Arthur DelhayeAffiliation:Computer Science, Université du Québec à Rimouski, CanadaYounghoon NaAffiliation:Department of Biomedical Sciences, Seoul National University College of Medicine, Republic of KoreaAffiliation:Obstructive Upper Airway Research \(OUaR\) Laboratory, Department of Pharmacology Seoul National University College of Medicine, Republic of KoreaHyun\-Woo ShinAffiliation:Department of Biomedical Sciences, Seoul National University College of Medicine, Republic of KoreaAffiliation:Obstructive Upper Airway Research \(OUaR\) Laboratory, Department of Pharmacology Seoul National University College of Medicine, Republic of KoreaAffiliation:Department of Otorhinolaryngology\-Head and Neck Surgery, Seoul National University Hospital, Republic of KoreaHyung\-Sin Kim†\\daggerAffiliation:Graduate School of Data Science, Seoul National University, South Korea

###### Abstract

Estimating full\-night electroencephalography \(EEG\) spectrograms from peripheral measurements could extend EEG\-informed sleep analysis to recordings acquired without EEG\. This task requires compact representations of signals with different temporal structures\. We introduceSleep2Word, a framework for translating respiration, electrocardiography \(ECG\), or photoplethysmography \(PPG\) into EEG spectrograms through duration\-aware physiological words\. Building on context\-dependent chunking,Sleep2Wordgroups physiological embeddings into variable\-duration chunks and jointly quantizes their content and duration into modality\-specific wordbooks\. A block\-autoregressive translator then predicts target EEG word identities and occurrence\-specific duration refinements within fixed\-duration blocks\. This design accommodates independent source and target wordbooks, limits temporal drift, and shortens the autoregressive decoding path\. Trained on over 20K PSG cases across 9 datasets,Sleep2Wordconsistently outperforms existing spectrogram generation baselines in EEG spectrogram reconstruction, sleep staging, arousal detection, and survival prediction\. Compressing full night sequences by up to 69\.7%, the learned words yield a favorable accuracy–compute trade\-off\. Sleep event analysis further shows that the learned words capture physiologically plausible associations across modalities, suggesting their potential for interpretable cross\-modal analysis\.

## 1Introduction

Electroencephalography \(EEG\) provides a detailed view of sleep\-related brain activity, but acquiring it routinely outside a sleep laboratory remains challenging\. Cardiac and respiratory measurements are more accessible and carry partial information about brain state through shared sleep physiology\([Jurysta et al\., 2003](https://arxiv.org/html/2611.21643#bib.bib4);[Schreiner et al\., 2023](https://arxiv.org/html/2611.21643#bib.bib3)\)\. We studyperipheral\-to\-EEG translation: estimating a full\-night EEG spectrogram from an individual peripheral source, respiration, electrocardiography \(ECG\), or photoplethysmography \(PPG\), without EEG at inference\. The target is spectral structure that can be predicted from the observed peripheral signal, not exact recovery of an unobserved EEG waveform\. The estimated spectrogram serves as a common intermediate output for multiple downstream analyses, rather than limiting the peripheral\-to\-brain mapping to a single task\-specific label\.

Physiology as Language\([Zha et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib12)\)established the feasibility of respiration\-to\-sleep\-EEG synthesis using continuous respiratory conditioning, a fixed grid of discrete EEG tokens, and masked generative prediction\. We build on this task by investigating how both source and target signals should be represented\. Fixed\-duration tokenization provides a convenient temporal grid but uses the same segmentation granularity throughout a recording\. Short intervals produce long full\-night sequences, whereas longer intervals provide a coarser representation\. Different physiological modalities also exhibit distinct temporal structures, motivating segmentations that can vary both within and across signals\. We therefore ask whether learned, variable\-duration segments can reduce the cost of full\-night translation while retaining the peripheral information needed to estimate EEG\. This motivatesadaptive representations on both sides of the translation, rather than prescribing one segment duration throughout each recording\.

Adaptive segmentation makes duration part of the translation problem\. The fifth respiratory token and the fifth EEG token need not describe the same recording interval, even when the underlying signals are synchronized\. Although source timestamps remain known, the decoder must predict both the content and duration of the target segments\. In a continuous autoregressive chain, accumulated duration errors can shift later predictions along the recording\. We therefore seek modality\-specific segmentations whose translation remains referenced to a common physical timeline, without requiring corresponding physiological patterns to occur simultaneously\.

![Refer to caption](https://arxiv.org/html/2611.21643v1/sleep2word_overview_v4.png)Figure 1:Modality\-specific physiological words on a shared recording timeline\.Sleep2Wordrepresents peripheral and EEG signals using independently learned, variable\-duration words\. EEG words are generated autoregressively within block, regularizing duration drift and enabling parallel generation of different blocks\.We introduceSleep2Word\(Figure[1](https://arxiv.org/html/2611.21643#S1.F1)\), guided by a principle: let each modality learn its own segments, but organize translation around common recording intervals\.Sleep2Wordadapts context\-dependent chunking\([Hwang et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib18)\)to physiological embeddings from SleepMaMi\([Park et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib11)\), grouping adjacent embeddings into variable\-duration chunks under reconstruction supervision\.Duration\-aware wordizationthen quantizes each chunk’s pooled representation together with its duration into a modality\-specific wordbook\. A physiological word is a learned discrete prototype of chunk content and temporal extent\. Chunking shortens the sequence, while wordization expresses translation as prediction over reusable segment codes\. Because occurrences can differ from a word’s typical duration, the decoder predicts an occurrence\-specific duration adjustment\.

To translate these words, a full\-night source encoder first captures context across the recording\. Ablock\-autoregressive decoderthen generates EEG words and their duration adjustments within fixed\-duration*blocks*, with sequential prediction inside each block and parallel generation across blocks\. The blocks organize generation in time, while the words retain adaptive segmentation within them\. Compared with an uninterrupted full\-night autoregressive chain, this design shortens the sequential decoding path and is intended to limit cumulative timing errors\. Tokenization and generation are therefore coordinated through a common temporal partition\.

Our contributions are threefold:

- •Duration\-aware physiological wordization\. We adapt context\-dependent chunking and jointly quantize chunk content and duration to construct modality\-specific wordbooks for peripheral\-to\-EEG translation\. The representation supports variable\-duration source and target sequences without imposing matching segmentation\.
- •Temporally anchored word\-to\-word translation\. We adapt block\-autoregressive decoding to physical recording intervals, complemented by occurrence\-specific duration prediction\. This confines temporal drift of variable\-length words within individual blocks and keeps the source and target aligned on the recording timeline\.
- •Evaluation across peripheral sources and downstream tasks\. We evaluate respiration\-, ECG\-, and PPG\-to\-EEG translation across SHHS1 and MESA as well as a held\-out dataset CFS\.Sleep2Wordoutperforms existing baselines in spectrogram reconstruction, downstream tasks including sleep staging, arousal detection and survival prediction, while demonstrating superior computational efficiency\.

## 2Related Work

### 2\.1Cross\-System Physiological Coupling and Cross\-Modal Modeling

Accumulating evidence indicates that central, autonomic, and respiratory processes are coupled across multiple timescales\. During sleep, respiration has been shown to modulate the timing of non\-rapid\-eye\-movement \(NREM\) slow oscillations and sleep spindles, such that these events preferentially occur at specific phases of the respiratory cycle\([Schreiner et al\., 2023](https://arxiv.org/html/2611.21643#bib.bib3)\)\. In addition, cortical slow oscillations are temporally coupled with transient heart\-rate bursts during sleep, demonstrating an interaction between central and autonomic activity\([Naji et al\., 2019](https://arxiv.org/html/2611.21643#bib.bib9)\)\. Together, these findings suggest that peripheral physiological signals retain information about brain states through shared physiological regulation\.

Physiological machine learning exploits cross\-system dependencies for representation learning, direct prediction, and signal generation\. Sleep foundation models learn reusable representations from multimodal recordings\([Park et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib11);[Thapa et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib10)\), while xMAE uses masked cross\-modal reconstruction as supervision for biosignal representation learning\([Zhou et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib14)\)\. Direct prediction methods such as wav2sleep\([Carter and Tarassenko, 2025](https://arxiv.org/html/2611.21643#bib.bib8)\)instead map physiological inputs to sleep\-stage labels\. Cross\-modal signal generation methods includes PPG\-to\-ECG\([Ding et al\., 2025](https://arxiv.org/html/2611.21643#bib.bib13)\), PPG\-to\-respiration\([Shuzan et al\., 2024](https://arxiv.org/html/2611.21643#bib.bib15)\), and respiration\-to\-EEG\([Zha et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib12)\)synthesis\. Most closely related, Physiology as Language\([Zha et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib12)\)combines continuous respiratory conditioning with a fixed temporal grid of discrete EEG codes\. Instead,Sleep2Wordrepresents both source and target throughmodality\-specific words\. It investigates whether this word\-based compact representation can support accurate cross\-modal estimation\.

### 2\.2Adaptive Chunking and Discrete Signal Representations

Adaptive sequence models reduce fine\-grained processing by forming input\-dependent segments\. SpaceByte\([Slagle, 2024](https://arxiv.org/html/2611.21643#bib.bib17)\)concentrates higher\-capacity processing around delimiter\-like boundaries, while BLT\([Pagnoni et al\., 2025](https://arxiv.org/html/2611.21643#bib.bib16)\)forms variable\-length byte patches using predictive entropy\. H\-Net\([Hwang et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib18)\)learns context\-dependent boundaries through a trainable routing module\.Sleep2Wordbuilds on this learned chunking mechanism and adapts it using physiological reconstruction and synchronization\-oriented boundary regularization\.

Discrete signal vocabularies have also been studied beyond language\. TFM\-Tokenizer\([Pradeepkumar et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib5)\)learns discrete time–frequency motifs from single\-channel EEG\. In speech, VARSTok\([Zheng et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib7)\)combines a content\-code index and a discrete duration into a unified token ID\. In contrast,Sleep2Wordmakes duration part of the quantization space, so content and temporal extent jointly determine each*word*in a fixed\-size wordbook\. On decoding, durations are then adjusted per occurrence rather than recovered exactly from the token ID\. We study this representation in peripheral\-to\-EEG translation, where the learned segmentation is additionally coordinated with the recording intervals used for decoding\.

### 2\.3Parallel Decoding and Temporal Organization

Semi\-autoregressive generation reduces sequential decoding while retaining target dependencies\.[Wang et al\. \(2018\)](https://arxiv.org/html/2611.21643#bib.bib36)generate token groups autoregressively, with parallel prediction within each group\. RecoverSAT\([Ran et al\., 2020](https://arxiv.org/html/2611.21643#bib.bib6)\)instead generates segments in parallel and tokens sequentially within each segment\.Sleep2Worduses a related schedule\. Fixed\-duration blocks are decoded in parallel, while EEG words are predicted sequentially within each block\. This keeps modality\-specific segmentations aligned to a common recording timeline\.

![Refer to caption](https://arxiv.org/html/2611.21643v1/sleep2word_method.png)Figure 2:Training pipeline\.Sleep2Wordis trained in three stages\. The full night sequence is segmented into context\-based chunks and discretized by modality\-specific wordbook\. The wordbook encodes both content and duration\. The full night word sequence is processed by an encoder, followed by block\-autoregressive decoder for temporal consistency\.

## 3Method

### 3\.1Overview and Problem Formulation

Our objective is to estimate an EEG spectrogram from peripheral physiological signals\. Let𝐱m∈ℝL\\mathbf\{x\}^\{m\}\\in\\mathbb\{R\}^\{L\}denote a full night recording from peripheral inputmm, whereL=fs​NsL=f\_\{s\}N\_\{s\},fsf\_\{s\}is the sampling rate andNsN\_\{s\}is the recording duration in seconds\. We transform the paired target EEG signal into a multi\-taper spectrogram\([Prerau et al\., 2017](https://arxiv.org/html/2611.21643#bib.bib30)\), denoted by𝐲∈ℝTy×Nf\\mathbf\{y\}\\in\\mathbb\{R\}^\{T\_\{y\}\\times N\_\{f\}\}whereTyT\_\{y\}is the number of non\-overlappingω\\omega\-second windows andNfN\_\{f\}is the number of frequency bins \(See Appendix[D](https://arxiv.org/html/2611.21643#A4)for details\)\. For clarity, we omit the modality superscript where the distinction is evident\.

Sleep2Wordestimates the target through a compact intermediate representation\. \(i\)*Context\-dependent chunking*first groups adjacent embeddings into variable\-duration segments\. \(ii\)*Wordization*maps each segment to a reusable content\-duration word\. \(iii\) The translator then predicts EEG word identities and their occurrence\-specific durations within common recording intervals, and the EEG reconstruction modules map the resulting sequence to a spectrogram\. The overall pipeline is depicted in Figure[2](https://arxiv.org/html/2611.21643#S2.F2)\.

#### Base embeddings\.

We construct a modality\-specific wordbooks using shared embeddings produced by the micro\-encoder of SleepMaMi\([Park et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib11)\), a recent sleep foundation model\. Whereas the original SleepMaMi forms shared representations by concatenating embeddings across all modalities, we modify its shared module to process each modality independently\. Consequently, the source signal can be encoded without access to the target modality\. Further details of this modification are provided in the Appendix[E](https://arxiv.org/html/2611.21643#A5)\.

### 3\.2Context\-Dependent Chunking

In the first stage, we learn to divide each full\-night representation sequence into variable\-length chunks whose boundaries depend on temporal context\. To this end, we adapt the dynamic chunking module of[Hwang et al\. \(2026\)](https://arxiv.org/html/2611.21643#bib.bib18)to physiological reconstruction\. Given a sequence of SleepMaMi embeddings𝐞∈ℝL′×d\\mathbf\{e\}\\in\\mathbb\{R\}^\{L^\{\\prime\}\\times d\}, whereL′L^\{\\prime\}is the number of base patches in the full night recording andddis the embedding dimension, the chunking module refines the embeddings and predicts the probabilitypip\_\{i\}that base positioniibegins a new chunk:

\(𝐳,𝐩\)=Chunk⁡\(𝐞\),𝐳∈ℝL′×d,𝐩∈\[0,1\]L′\.\(\\mathbf\{z\},\\mathbf\{p\}\)=\\mathrm\{Chunk\}\(\\mathbf\{e\}\),\\quad\\mathbf\{z\}\\in\\mathbb\{R\}^\{L^\{\\prime\}\\times d\},\\quad\\mathbf\{p\}\\in\[0,1\]^\{L^\{\\prime\}\}\.\(1\)During the forward pass, the start positions are\{1\}∪\{i≥2andpi≥0\.5\}\\\{1\\\}\\cup\\\{i\\geq 2\\text\{ and \}p\_\{i\}\\geq 0\.5\\\}\. Lets1<⋯<sCs\_\{1\}<\\cdots<s\_\{C\}denote the resulting start positions, withsC\+1=L′\+1s\_\{C\+1\}=L^\{\\prime\}\+1\. The representation of thecc\-th chunk is obtained by average pooling:

𝐳¯c=1lc​∑t=scsc\+lc−1𝐳t,lc=sc\+1−sc\.\\bar\{\\mathbf\{z\}\}\_\{c\}=\\frac\{1\}\{l\_\{c\}\}\\sum\_\{t=s\_\{c\}\}^\{s\_\{c\}\+l\_\{c\}\-1\}\\mathbf\{z\}\_\{t\},\\quad l\_\{c\}=s\_\{c\+1\}\-s\_\{c\}\.\(2\)The pair\(sc,lc\)\(s\_\{c\},l\_\{c\}\)is retained alongside each pooled representation to preserve its position and temporal length\. The chunk representations are subsequently scattered back to their original positions:

𝐮t=𝐳¯cforsc≤t<sc\+1\\mathbf\{u\}\_\{t\}=\\bar\{\\mathbf\{z\}\}\_\{c\}\\quad\\text\{for\}\\quad s\_\{c\}\\leq t<s\_\{c\+1\}\(3\)The resulting sequence is processed by the dechunking module and a convolutional reconstruction head, which recovers the original waveform for peripheral modalities and the corresponding spectrogram for EEG\. We use mean squared error as reconstruction objective \(ℒrecon\\mathcal\{L\}\_\{\\mathrm\{recon\}\}\)\.

Because hard thresholding prevents reconstruction gradients from propagating through the boundary decisions, we apply a straight\-through surrogate by replacing𝐮t\\mathbf\{u\}\_\{t\}with𝐮~t=𝐮t​\(1\+\(psc−sg⁡\[psc\]\)\)\\tilde\{\\mathbf\{u\}\}\_\{t\}=\\mathbf\{u\}\_\{t\}\(1\+\(p\_\{s\_\{c\}\}\-\\mathrm\{sg\}\[p\_\{s\_\{c\}\}\]\)\)wheresg​\[⋅\]\\text\{sg\}\[\\cdot\]denotes the stop\-gradient operator\. This does not influence the forward pass, but allows the gradient to reach𝐩\\mathbf\{p\}during backpropagation\.

In addition to the reconstruction objective, we introduce three auxiliary losses\. First, following\([Hwang et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib18)\), we adopt a ratio loss \(ℒratio\\mathcal\{L\}\_\{\\mathrm\{ratio\}\}\) to regulate the proportion of predicted boundaries\. Second, an alignment loss \(ℒalign\\mathcal\{L\}\_\{\\mathrm\{align\}\}\) anchors the refined embeddings to the original shared embeddings, preventing them from deviating substantially toward modality\-specific representation space\. Finally, the barrier loss \(ℒbarrier\\mathcal\{L\}\_\{\\mathrm\{barrier\}\}\) encourages boundaries to start at the common recording intervals \(BBbase positions\)\. This constraint promotes periodic synchronization between the source and target sequences, facilitating the block autoregressive translation described in Section[3\.4](https://arxiv.org/html/2611.21643#S3.SS4)\. The details of these auxiliary losses are provided in Appendix[F](https://arxiv.org/html/2611.21643#A6)\.

The overall chunking objective is defined as follows:

ℒchunk\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{chunk\}\}=ℒrecon\+λratio​ℒratio\+λalign​ℒalign\+λbarrier​ℒbarrier\\displaystyle=\\mathcal\{L\}\_\{\\mathrm\{recon\}\}\+\\lambda\_\{\\mathrm\{ratio\}\}\\mathcal\{L\}\_\{\\mathrm\{ratio\}\}\+\\lambda\_\{\\mathrm\{align\}\}\\mathcal\{L\}\_\{\\mathrm\{align\}\}\+\\lambda\_\{\\mathrm\{barrier\}\}\\mathcal\{L\}\_\{\\mathrm\{barrier\}\}\(4\)

### 3\.3Wordization

We construct a modality\-specific wordbook𝒲=\{wk\}k=1K\\mathcal\{W\}=\\\{w\_\{k\}\\\}\_\{k=1\}^\{K\}from the learned chunk representations\. To incorporate temporal extent into each word, we augment the pooled representation with log duration:𝐳c′=\[𝐳¯sc;α​log​\(lc\)\]\\mathbf\{z\}^\{\\prime\}\_\{c\}=\[\\bar\{\\mathbf\{z\}\}\_\{s\_\{c\}\};\\alpha\\text\{log\}\(l\_\{c\}\)\]whereα\\alphacontrols the scale of duration relative to content\. We represent each wordbook entry aswk=\[νk;ηk\]w\_\{k\}=\[\\nu\_\{k\};\\eta\_\{k\}\], whereνk∈ℝd\\nu\_\{k\}\\in\\mathbb\{R\}^\{d\}represents its content andηk∈ℝ\\eta\_\{k\}\\in\\mathbb\{R\}represents its duration component\. Each chunk𝐳c′\\mathbf\{z\}^\{\\prime\}\_\{c\}is assigned to its nearest wordbook entry following this rule:

kc=arg⁡mink∈\{1,…,K\}⁡\[‖νk−𝐳¯c‖22\+\(α​log⁡lc−ηk\)2\]k\_\{c\}=\\arg\\min\_\{k\\in\\\{1,\.\.\.,K\\\}\}\[\\\|\\mathbf\{\\nu\}\_\{k\}\-\\bar\{\\mathbf\{z\}\}\_\{c\}\\\|\_\{2\}^\{2\}\+\(\\alpha\\log l\_\{c\}\-\\eta\_\{k\}\)^\{2\}\]\(5\)wherekck\_\{c\}is the discrete word index\. This formulation helps both content and duration to contribute directly to word identity\. In addition, for each word, we maintain an exponential moving averageμk\\mu\_\{k\}of the durations of its assigned chunks for use in subsequent duration prediction\.

During wordization, the chunking module is frozen\. Each chunk is assigned to a word indexkck\_\{c\}and the corresponding word embedding is passed through the dechunking module and reconstruction head to recover the original peripheral signal or EEG spectrogram\. The wordization objective is

ℒWordization=ℒr​e​c​o​n\+λQC​∑c=1C‖sg⁡\[𝐳c′\]−𝐰kc‖22\\mathcal\{L\}\_\{\\mathrm\{Wordization\}\}=\\mathcal\{L\}\_\{recon\}\+\\frac\{\\lambda\_\{Q\}\}\{C\}\\sum\_\{c=1\}^\{C\}\\\|\\mathrm\{sg\}\[\\mathbf\{z\}^\{\\prime\}\_\{c\}\]\-\\mathbf\{w\}\_\{k\_\{c\}\}\\\|\_\{2\}^\{2\}\(6\)With fixed chunk representations and fixedα\\alpha, we do not include the term‖𝐳c′−sg​\[𝐰kc\]‖22\\\|\\mathbf\{z\}^\{\\prime\}\_\{c\}\-\\text\{sg\}\[\\mathbf\{w\}\_\{k\_\{c\}\}\]\\\|\_\{2\}^\{2\}, unlike conventional vector quantization\([Van Den Oord et al\., 2017](https://arxiv.org/html/2611.21643#bib.bib31)\)\.

We initialize the wordbook and calculateα\\alphausing statistics from the firstIIbatches \(We useI=10I=10\) produced by the frozen chunking module\. The calibration ofα\\alphabalances the contributions of content and log\-duration to word assignment, encouraging sensitivity to the chunk duration without requiring a distinct word for every possible length\. More details of wordbook initialization are provided in Appendix[C](https://arxiv.org/html/2611.21643#A3)\.

### 3\.4Peripheral to EEG translation

After constructing separate wordbooks for the source and target modalities, we train an encoder\-decoder model to translate a sequence of peripheral words into a sequence of EEG words, with chunking, dechunking and reconstruction modules frozen\. Before target generation begins, the encoder processes the complete peripheral word sequence\{𝐰kcm\}c=1Cm\\\{\\mathbf\{w\}\_\{k\_\{c\}\}^\{m\}\\\}\_\{c=1\}^\{C\_\{m\}\}\. Rotary positional embeddings[Su et al\. \(2024\)](https://arxiv.org/html/2611.21643#bib.bib35)encode the original start position of each source word, retaining recording time information despite variable word durations\. The encoder consists of four Transformer blocks[Vaswani et al\. \(2017\)](https://arxiv.org/html/2611.21643#bib.bib34), each containing two sliding window self\-attention layers\([Beltagy et al\., 2020](https://arxiv.org/html/2611.21643#bib.bib33)\)followed by one full\-attention layer\. This architecture captures both local temporal patterns and full\-night source context\.

The decoder generates target EEG words conditioned on the encoded source sequence\. Because the source and target modalities are chunked independently, their word sequences may differ in both the number and duration of their words\. Naïve word\-to\-word decoding therefore accumulates temporal misalignment over the course of a full\-night recording\. To constrain this drift, we employBlock AutoRegressive \(AR\)decoding\. Let𝐇bm\\mathbf\{H\}^\{m\}\_\{b\}denote the encoded source memory associated with thebb\-th block\. Within each block, the decoder generates EEG words autoregressively while conditioning on𝐇bm\\mathbf\{H\}^\{m\}\_\{b\}\. This formulation bounds any temporal discrepancy within individual blocks and reduces the sequential decoding path relative to full\-night AR generation\.

LetNB=⌈L′/B⌉N\_\{B\}=\\left\\lceil L^\{\\prime\}/B\\right\\rceilbe the number of blocks andnbn\_\{b\}the number of target words in blockbb\. For training labelskb,sE​E​Gk\_\{b,s\}^\{EEG\}obtained from frozen EEG tokenizer, the word\-identity objective is

ℒid=−1NB∑b=1NB1nb∑s=1nblogp\(kb,sE​E​G\|kb,<sE​E​G,𝐇bm\)\\mathcal\{L\}\_\{\\mathrm\{id\}\}=\-\\frac\{1\}\{N\_\{B\}\}\\sum\_\{b=1\}^\{N\_\{B\}\}\\frac\{1\}\{n\_\{b\}\}\\sum\_\{s=1\}^\{n\_\{b\}\}\\log p\(k\_\{b,s\}^\{EEG\}\|k\_\{b,<s\}^\{EEG\},\\mathbf\{H\}^\{m\}\_\{b\}\)\(7\)
Occurrence\-specific duration\. In addition to word identity, the decoder predicts a duration residualr^b,s\\hat\{r\}\_\{b,s\}relative to the EMA of the selected target word\. Each word represents a prototypical content\-duration pattern, and the residual accommodates variation across individual occurrences\. Using target tokenizer residualrb,s=lb,sE​E​G−round⁡\(μkb,s\)r\_\{b,s\}=l\_\{b,s\}^\{EEG\}\-\\mathrm\{round\}\(\\mu\_\{k\_\{b,s\}\}\)as supervision, the duration and overall translation loss are defined as

ℒdur=−1NB∑b=1NB1nb∑s=1nblogp\(rb,sE​E​G\|kb,sE​E​G,𝐇bm\),ℒtrans=ℒid\+λdurℒdur\\mathcal\{L\}\_\{\\mathrm\{dur\}\}=\-\\frac\{1\}\{N\_\{B\}\}\\sum\_\{b=1\}^\{N\_\{B\}\}\\frac\{1\}\{n\_\{b\}\}\\sum\_\{s=1\}^\{n\_\{b\}\}\\log p\(r\_\{b,s\}^\{EEG\}\|k\_\{b,s\}^\{EEG\},\\mathbf\{H\}^\{m\}\_\{b\}\),\\quad\\mathcal\{L\}\_\{\\mathrm\{trans\}\}=\\mathcal\{L\}\_\{\\mathrm\{id\}\}\+\\lambda\_\{\\mathrm\{dur\}\}\\mathcal\{L\}\_\{\\mathrm\{dur\}\}\(8\)
Training\. Throughout translation training, the chunking, dechunking and reconstruction modules remain frozen\. Training begins with teacher\-forcing after which scheduled sampling\([Bengio et al\., 2015](https://arxiv.org/html/2611.21643#bib.bib32)\)is applied with a progressively increasing probability to mitigate exposure bias\.

Inference and reconstruction\. During inference, the target words are generated within each block until their cumulative predicted duration reaches the corresponding block length in base positions\. For blockbb, this target length isTb=min⁡\(B,L′−\(b−1\)​B\)T\_\{b\}=\\min\\left\(B,L^\{\\prime\}\-\(b\-1\)B\\right\)\. If the cumulative duration exceedsTbT\_\{b\}, the duration of the final word is truncated\. If the generated sequence length falls short, the duration of the final word is extended to fill the remaining positions\. The predicted EEG word embeddings are then expanded onto the base grid using the predicted durations and processed by the frozen EEG dechunker and reconstruction head to obtain the full night EEG spectrogram\.

## 4Experiments

### 4\.1Datasets, Preprocessing and Baselines

We train on a combined set of nine publicly available sleep datasets: the Sleep Heart Health Study \(SHHS1/2\)\([Zhang et al\., 2018](https://arxiv.org/html/2611.21643#bib.bib21);[Quan et al\., 1997](https://arxiv.org/html/2611.21643#bib.bib22)\), Korea Image\-based Sleep Study \(KISS\)\([Jeong et al\., 2023](https://arxiv.org/html/2611.21643#bib.bib28)\), Korea Video Sleep Study \(KVSS\)\([Choi et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib27)\), Physionet 2018 \(PHY\)\([Goldberger et al\., 2000](https://arxiv.org/html/2611.21643#bib.bib19);[Ghassemi et al\., 2018](https://arxiv.org/html/2611.21643#bib.bib20)\), Multi\-Ethnic Study of Atherosclerosis \(MESA\)\([Chen et al\., 2015](https://arxiv.org/html/2611.21643#bib.bib23)\)and Osteoporotic Fractures in Men Study \(MrOS\)\([Blackwell et al\., 2011](https://arxiv.org/html/2611.21643#bib.bib24)\)\. Overall, we use 20,036 PSG recordings \(107,469 hours\) for training\. We use the Cleveland Family Study \(CFS\)\([Zhang et al\., 2018](https://arxiv.org/html/2611.21643#bib.bib21);[Redline et al\., 1995](https://arxiv.org/html/2611.21643#bib.bib29)\)as a held\-out dataset to assessSleep2Word’s generalizability to previously unseen data\. Further dataset details are provided in Appendix[A](https://arxiv.org/html/2611.21643#A1)\.

We apply standard preprocessing to all modalities\. EEG signals are filtered using a 0\.3–40 Hz bandpass filter, and ECG signals undergo a 0\.3–60 Hz bandpass filtering\. A 60 Hz notch filter is additionally applied to both EEG and ECG signals\. Respiratory signals are low\-pass filtered at 15 Hz\. Biosignals are resampled at 100 Hz to harmonize the differing sampling rates across the diverse datasets\. We use C4\-A1 EEG channel and abdominal belt respiratory signal\. For PPG, the raw signal is polarity\-corrected, low\-pass filtered at 8 Hz and clipped at±3\\pm 3standard deviations\. Before model input, all signals are z\-score normalized independently within each 30\-second epoch\.

As a baseline, we construct a simple CNN\-Transformer model for peripheral\-to\-EEG conversion \(referred to C\-T baseline, details in Appendix[B](https://arxiv.org/html/2611.21643#A2)\)\. For respiratory\-to\-EEG conversion, we compare against the recently proposed Physiology as Language \(PasL\)\([Zha et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib12)\)\. We reproduce PasL following the paper, since the source code is not released\. wav2sleep\([Carter and Tarassenko, 2025](https://arxiv.org/html/2611.21643#bib.bib8)\), a sleep staging specialized model using peripheral inputs, is also adopted to compare against raw signal based method\.

Baseline split alignment\.For SHHS1, MESA and CFS, we align our evaluation sets with the official wav2sleep splits to ensure comparisons are performed on the same recordings\. Details of the split construction are provided in Appendix[A](https://arxiv.org/html/2611.21643#A1)\.

### 4\.2EEG Reconstruction Performance

Table[1](https://arxiv.org/html/2611.21643#S4.T1)provides the EEG spectrogram reconstruction quality ofSleep2Wordand other baselines\. Following PasL, we report the quality using two metrics: Mean Absolute Error \(MAE\) and Signal\-to\-Noise Ratio \(SNR\), in comparison to the ground truth spectrogram\. Across all source modality inputs and datasets,Sleep2Wordexhibits lower MAE and higher SNR compared to other baselines\. Particularly,Sleep2Worddemonstrates its robustness to the unseen dataset \(CFS\), while PasL shows suboptimal performance\.

Table 1:EEG spectrogram reconstruction performance\. MAE and SNR are provided across modalities and datasets\. Bold values indicate the best among spectrogram\-generation methods\.Table 2:Sleep staging performance\. Best spectrogram\-generation results are bolded\.
### 4\.3Downstream Performance

We evaluate whether the generated EEG spectrograms preserve information relevant to three downstream tasks: four\-stage sleep staging, arousal detection, and survival prediction\.

Sleep staging\. Table[2](https://arxiv.org/html/2611.21643#S4.T2)reports performance on four\-stage sleep staging \(Wake, N1/N2, N3, and REM\) using Macro\-F1, accuracy, and Cohen’s kappa\. For spectrogram\-generation methods \(Sleep2Word, PasL, and the C\-T baseline\), we froze the generation model and separately fine\-tuned CNN classifier on spectrograms from the training split and evaluate it on the corresponding test split of each dataset\. For ground\-truth \(GT\) EEG spectrograms, a CNN classifier is also fine\-tuned on training split and evaluated\. For wav2sleep, we directly evaluate the released pretrained models\.

Among the evaluated spectrogram\-generation methods,Sleep2Wordachieves the strongest sleep staging performance across the datasets and source modalities\. WithinSleep2Word, respiration consistently yields better performance than ECG or PPG, suggesting that sleep\-stage\-related EEG structure can be recovered more reliably from respiratory signals in the evaluated cohorts\.

Nonetheless,Sleep2Wordgenerally underperforms wav2sleep and the classifier trained on GT EEG spectrograms\. This comparison should be interpreted in light of their different objectives: wav2sleep is optimized specifically for sleep staging using direct stage supervision, whereasSleep2Wordlearns task\-agnostic cross\-modal generation and is evaluated using a separate downstream classifier\. The remaining gap to GT EEG further suggests that some stage\-relevant information is either unavailable in peripheral signals or lost during word\-based generation\. Despite these differences, the consistent improvement over other spectrogram\-generation baselines demonstrates thatSleep2Wordpreserves more stage\-discriminative EEG structure without task\-specific optimization\.

Arousal detection\. Table[3](https://arxiv.org/html/2611.21643#S4.T3)compares the arousal detection performance ofSleep2Wordwith those of the baseline models\. To match the temporal resolution of the input spectrograms, we convert the second\-level arousal annotations into epoch\-level labels\. An epoch is labeled as positive if it contains at least three seconds of annotated arousal activity\. Using the epoch\-level arousal labels, we train a linear classifier on the generated spectrogram\. The arousal prediction AUROC conditioned on non\-wake epochs is reported, to align with the arousal determination rule\([Berry, 2012](https://arxiv.org/html/2611.21643#bib.bib1)\)\.

Sleep2Wordachieves the highest AUROC among spectrogram\-generation methods across datasets and modalities\. In particular, it consistently outperforms wav2sleep\. Since wav2sleep is trained for sleep staging, this comparison indicates that its representations carry little arousal information beyond what staging requires\. This supports the use of generated spectrograms as a task\-agnostic intermediate rather than a proxy for a single label\.

Figure 3:Compute efficiency\.Sleep staging accuracy versus cumulative training compute\.Sleep2Wordachieves the best accuracy\-compute trade\-off among the compared methodsSurvival prediction\. We evaluate prognostic information in generated EEG spectrograms on SHHS1 using a linear Cox head for all\-cause mortality, CVD mortality, and stroke\.Sleep2Wordachieves higher C\-indices than PasL and C\-T across all respiratory\-input outcomes, and exceeds wav2sleep for CVD mortality while remaining comparable for the other outcomes\. With ECG input,Sleep2Wordachieves higher C\-indices than C\-T and wav2sleep across all outcomes \(Table[4](https://arxiv.org/html/2611.21643#S4.T4)\)\.

Table 3:Arousal detection performance\. Results are provided in AUROC conditioned on non\-wake epochs\. Bold values indicate the best performance among spectrogram\-generation methods\.Table 4:Survival prediction performance on SHHS1\. Results are reported using the C\-index\. Bold values indicate the best performance among spectrogram\-generation methods\.
### 4\.4Compute efficiency

To assess the computational efficiency ofSleep2Word, we compare its cumulative training compute, measured in TFLOPs, with linear probed sleep staging accuracy on SHHS1\. Here, we introduce Base2base, which directly translates a sequence of source\-modality base patch embeddings𝐞m\\mathbf\{e\}^\{m\}, into the corresponding sequence of EEG base patch embeddings𝐞E​E​G\\mathbf\{e\}^\{EEG\}, without intermediate chunking or wordization\. As shown in Figure[3](https://arxiv.org/html/2611.21643#S4.F3),Sleep2Wordachieves higher accuracy than the baselines at comparable or lower compute budgets\. This result demonstrates the superior accuracy\-compute trade\-off enabled by the word\-based framework\.

### 4\.5Qualitative Analysis and Ablation

Figures[4\(a\)](https://arxiv.org/html/2611.21643#S4.F4.sf1)and[4\(b\)](https://arxiv.org/html/2611.21643#S4.F4.sf2)compare representative EEG spectrograms generated from peripheral signals with their corresponding ground\-truth spectrograms\. The generated spectrograms reproduce broad full\-night spectral structure, including variations in low frequency, spindle frequency range, and higher frequency power, although they do not capture all fine\-grained structures present in the ground\-truth EEG\. To examine whether the model captures cross\-modal physiological relationships, we investigate apnea\-arousal coupling\. We identify the 20 EEG words \(the top\-1% of wordbook\) most strongly associated with annotated arousals and provide respiratory input only until apnea termination, while allowing EEG generation to continue beyond the event\. Figure[4\(c\)](https://arxiv.org/html/2611.21643#S4.F4.sf3)shows that the probability of arousal\-related EEG words increases sharply after apnea termination and exceeds those under random\-state conditioning\. The example in Figure[4\(d\)](https://arxiv.org/html/2611.21643#S4.F4.sf4)further illustrates this pattern, suggesting that the model learns the temporal relationship between disordered breathing and subsequent cortical arousal\. Together, these results highlight the potential of word\-based approach for model\-based exploration of cross\-modal physiological associations\.

Ablation results \(Appendix[G](https://arxiv.org/html/2611.21643#A7)\) confirm the contribution of each core component: loss components \(Table[13](https://arxiv.org/html/2611.21643#A7.T13)\), chunking and wordization \(Table[15](https://arxiv.org/html/2611.21643#A7.T15)\), duration modeling \(Table[15](https://arxiv.org/html/2611.21643#A7.T15)\), wordbook size \(Table[18](https://arxiv.org/html/2611.21643#A7.T18)\), target chunking ratio \(Table[18](https://arxiv.org/html/2611.21643#A7.T18)\) and block size \(Table[18](https://arxiv.org/html/2611.21643#A7.T18)\)\.

![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/spec_RESP_shhs1_shhs1-205246_M60.png)\(a\)Spectrogram \(from Resp\.\): 60\-year\-old Male
![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/spec_ECG_kiss_A2019-NX-01-0843_M54.png)\(b\)Spectrogram \(from ECG\): 54\-year\-old Male
![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/probe_steps_compact.png)\(c\)EEG arousal\-word probabilities on apnea input
![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/shhs1-200001_Apnea-Hypopnea_5205s_compact.png)\(d\)Arousal generation example after apnea

Figure 4:Qualitative analysis\. \(a\-b\) Full\-night comparison between the generated and ground\-truth EEG spectrograms\. \(c\) Changes in the probability of generating arousal\-related EEG words after apnea termination\. \(d\) An example showing the input respiratory signal from respiratory event, generated EEG words, and spectrogram\. The bolded box represents an arousal word\.

## 5Conclusion

In this paper, we presentSleep2Word, a word\-based framework for peripheral\-to\-EEG translation\. By introducing context\-dependent chunking, duration\-aware physiological wordization, and a block\-AR decoder, our model captures the distinct temporal structures of different modalities while effectively mitigating temporal drift\. Our evaluations confirm thatSleep2Wordoutperforms existing baselines in EEG spectrogram reconstruction and clinical downstream tasks\. Finally, the word\-based architecture reduces the sequence length by 69\.7%, enabling a favorable compute\-accuracy trade\-off\. This work provides a strong foundation for expanding sleep neurophysiology analysis using more accessible peripheral measurements\.

### AI use statement

In this work, we used generative AI tools to provide feedback on research methodology or experiments, and implement methods\. We have not used generative AI tools for other tasks with required disclosure, and they are not applicable to this work\. Additionally, we used generative AI tools to create or edit software code, summarize or analyze existing literature and edit a research paper to improve readability\. We have reviewed all AI\-assisted work\. For example, we manually checked the ideas, sentences, existing literature suggested by AI for its factual accuracy and potential plagiarism\. LLM\-generated code was also verified and tested for correctness by 2 authors\. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.

### Reproducibility statement

To ensure the reproducibility of our work, we provide the model specification \(Appendix[B](https://arxiv.org/html/2611.21643#A2)\) and implementation details \(Appendix[C](https://arxiv.org/html/2611.21643#A3)\)\. The summary information on datasets \(Appendix[A](https://arxiv.org/html/2611.21643#A1)\) and preprocessing steps \(Section[4](https://arxiv.org/html/2611.21643#S4)\) are also provided\.

## References

- Ahnet al\.\(2026\)H\. Ahn, Y\. Na, and H\. ShinRefining sleep‐disordered breathing annotations across multiple public sleep study datasets\.Journal of Sleep Research35,pp\.\.External Links:[Document](https://dx.doi.org/10.1111/jsr.70264)Cited by:[§A\.5](https://arxiv.org/html/2611.21643#A1.SS5.p1.1),[§A\.6](https://arxiv.org/html/2611.21643#A1.SS6.p1.1)\.
- AIHub \(2020\)AIHubAIHub\. image of sleep quality assessment and sleep disorder diagnosis\.\.Note:[https://www\.aihub\.or\.kr/aihubdata/data/view\.do?currMenu=&topMenu=&aihubDataSe=realm&dataSetSn=210](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=&topMenu=&aihubDataSe=realm&dataSetSn=210)Cited by:[§A\.3](https://arxiv.org/html/2611.21643#A1.SS3.p1.1)\.
- Beltagyet al\.\(2020\)I\. Beltagy, M\. E\. Peters, and A\. CohanLongformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.Cited by:[§3\.4](https://arxiv.org/html/2611.21643#S3.SS4.p1.1)\.
- Bengioet al\.\(2015\)S\. Bengio, O\. Vinyals, N\. Jaitly, and N\. ShazeerScheduled sampling for sequence prediction with recurrent neural networks\.Advances in neural information processing systems28\.Cited by:[§3\.4](https://arxiv.org/html/2611.21643#S3.SS4.p5.1)\.
- Berry \(2012\)R\. BerryThe aasm manual for the scoring of sleep and associated events\.Rules, Terminology and Technical Specifications\. Version 22\.Cited by:[§A\.3](https://arxiv.org/html/2611.21643#A1.SS3.p1.1),[§4\.3](https://arxiv.org/html/2611.21643#S4.SS3.p5.1)\.
- Berryet al\.\(2017\)R\. B\. Berry, R\. Brooks, C\. Gamaldo, S\. M\. Harding, R\. M\. Lloyd, S\. F\. Quan, M\. T\. Troester, and B\. V\. VaughnAASM scoring manual updates for 2017 \(version 2\.4\)\.Journal of Clinical Sleep Medicine13\(5\),pp\. 665–666\.Cited by:[§A\.3](https://arxiv.org/html/2611.21643#A1.SS3.p1.1)\.
- Blackwellet al\.\(2011\)T\. Blackwell, K\. Yaffe, S\. Ancoli\-Israel, S\. Redline, K\. E\. Ensrud, M\. L\. Stefanick, A\. Laffan, K\. L\. Stone, and O\. F\. in Men Study GroupAssociations between sleep architecture and sleep\-disordered breathing and cognition in older community\-dwelling men: the osteoporotic fractures in men sleep study\.Journal of the American Geriatrics Society59\(12\),pp\. 2217–2225\.Cited by:[§A\.6](https://arxiv.org/html/2611.21643#A1.SS6.p1.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p1.1)\.
- Carter and Tarassenko \(2025\)J\. F\. Carter and L\. TarassenkoWav2sleep: a unified multi\-modal approach to sleep stage classification from physiological signals\.InProceedings of the 4th Machine Learning for Health Symposium,Proceedings of Machine Learning Research, Vol\.259,pp\. 186–202\.Cited by:[§2\.1](https://arxiv.org/html/2611.21643#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p3.1)\.
- Chenet al\.\(2015\)X\. Chen, R\. Wang, P\. Zee, P\. L\. Lutsey, S\. Javaheri, C\. Alcántara, C\. L\. Jackson, M\. A\. Williams, and S\. RedlineRacial/ethnic differences in sleep disturbances: the multi\-ethnic study of atherosclerosis \(mesa\)\.Sleep38\(6\),pp\. 877–888\.Cited by:[§A\.5](https://arxiv.org/html/2611.21643#A1.SS5.p1.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p1.1)\.
- Choiet al\.\(2026\)Y\. R\. Choi, D\. Park, H\. K\. Lee, S\. Bae, S\. Park, D\. Kim, Y\. M\. Kim, C\. Park, H\. Kim, and H\. ShinNon\-contact on\-device detection of obstructive sleep apnea from infrared video\.Nature Communications\.Cited by:[§A\.4](https://arxiv.org/html/2611.21643#A1.SS4.p1.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p1.1)\.
- Dinget al\.\(2025\)Z\. Ding, Y\. Hu, Z\. Li, Y\. Mao, H\. Li, D\. Zhou, X\. Chu, L\. Yu, Z\. Liu, F\. Wu,et al\.AI modeling photoplethysmography to electrocardiography useful for predicting cardiovascular disease\.NPJ Digital Medicine9\(1\),pp\. 61\.Cited by:[§2\.1](https://arxiv.org/html/2611.21643#S2.SS1.p2.1)\.
- Ghassemiet al\.\(2018\)M\. M\. Ghassemi, B\. E\. Moody, L\. H\. Lehman, C\. Song, Q\. Li, H\. Sun, R\. G\. Mark, M\. B\. Westover, and G\. D\. CliffordYou snooze, you win: the physionet/computing in cardiology challenge 2018\.In2018 Computing in Cardiology Conference,Vol\.45,pp\. 1–4\.Cited by:[§A\.1](https://arxiv.org/html/2611.21643#A1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p1.1)\.
- Goldbergeret al\.\(2000\)A\. L\. Goldberger, L\. A\. Amaral, L\. Glass, J\. M\. Hausdorff, P\. C\. Ivanov, R\. G\. Mark, J\. E\. Mietus, G\. B\. Moody, C\. Peng, and H\. E\. StanleyPhysioBank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals\.Circulation101\(23\),pp\. e215–e220\.Cited by:[§A\.1](https://arxiv.org/html/2611.21643#A1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p1.1)\.
- Hwanget al\.\(2026\)S\. Hwang, B\. Wang, and A\. GuDynamic chunking for end\-to\-end hierarchical sequence modeling\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 149273–149313\.Cited by:[Appendix F](https://arxiv.org/html/2611.21643#A6.p2.1),[§1](https://arxiv.org/html/2611.21643#S1.p4.1),[§2\.2](https://arxiv.org/html/2611.21643#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2611.21643#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2611.21643#S3.SS2.p3.1)\.
- Jeonget al\.\(2023\)J\. Jeong, W\. Yoon, J\. Lee, D\. Kim, Y\. Woo, D\. Kim, and H\. ShinStandardized image\-based polysomnography database and deep learning algorithm for sleep\-stage classification\.Sleep46\(12\),pp\. zsad242\.Cited by:[§A\.3](https://arxiv.org/html/2611.21643#A1.SS3.p1.1),[§A\.3](https://arxiv.org/html/2611.21643#A1.SS3.p2.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p1.1)\.
- Jurystaet al\.\(2003\)F\. Jurysta, P\. Van De Borne, P\. Migeotte, M\. Dumont, J\. Lanquart, J\. Degaute, and P\. LinkowskiA study of the dynamic interactions between sleep eeg and heart rate variability in healthy young men\.Clinical neurophysiology114\(11\),pp\. 2146–2155\.Cited by:[§1](https://arxiv.org/html/2611.21643#S1.p1.1)\.
- Najiet al\.\(2019\)M\. Naji, G\. P\. Krishnan, E\. A\. McDevitt, M\. Bazhenov, and S\. C\. MednickCoupling of autonomic and central events during sleep benefits declarative memory consolidation\.Neurobiology of learning and memory157,pp\. 139–150\.Cited by:[§2\.1](https://arxiv.org/html/2611.21643#S2.SS1.p1.1)\.
- Pagnoniet al\.\(2025\)A\. Pagnoni, R\. Pasunuru, P\. Rodriguez, J\. Nguyen, B\. Muller, M\. Li, C\. Zhou, L\. Yu, J\. E\. Weston, L\. Zettlemoyer,et al\.Byte latent transformer: patches scale better than tokens\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9238–9258\.Cited by:[§2\.2](https://arxiv.org/html/2611.21643#S2.SS2.p1.1)\.
- Parket al\.\(2026\)K\. Park, Y\. Na, Y\. R\. Choi, H\. Ryu, H\. Shin, and H\. KimSleepMaMi: a universal sleep foundation model for integrating macro\- and micro\-structures\.InForty\-third International Conference on Machine Learning,Cited by:[Appendix E](https://arxiv.org/html/2611.21643#A5.p1.1),[§1](https://arxiv.org/html/2611.21643#S1.p4.1),[§2\.1](https://arxiv.org/html/2611.21643#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2611.21643#S3.SS1.SSS0.Px1.p1.1)\.
- Pradeepkumaret al\.\(2026\)J\. Pradeepkumar, X\. Piao, Z\. Chen, and J\. SunTokenizing single\-channel eeg with time\-frequency motif learning\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 36611–36637\.Cited by:[§2\.2](https://arxiv.org/html/2611.21643#S2.SS2.p2.1)\.
- Prerauet al\.\(2017\)M\. J\. Prerau, R\. E\. Brown, M\. T\. Bianchi, J\. M\. Ellenbogen, and P\. L\. PurdonSleep neurophysiological dynamics through the lens of multitaper spectral analysis\.Physiology32\(1\),pp\. 60–92\.Cited by:[Appendix D](https://arxiv.org/html/2611.21643#A4.p2.1),[§3\.1](https://arxiv.org/html/2611.21643#S3.SS1.p1.1)\.
- Quanet al\.\(1997\)S\. F\. Quan, B\. V\. Howard, C\. Iber, J\. P\. Kiley, F\. J\. Nieto, G\. T\. O’Connor, D\. M\. Rapoport, S\. Redline, J\. Robbins, J\. M\. Samet,et al\.The sleep heart health study: design, rationale, and methods\.Sleep20\(12\),pp\. 1077–1085\.Cited by:[§A\.2](https://arxiv.org/html/2611.21643#A1.SS2.p1.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p1.1)\.
- Ranet al\.\(2020\)Q\. Ran, Y\. Lin, P\. Li, and J\. ZhouLearning to recover from multi\-modality errors for non\-autoregressive neural machine translation\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 3059–3069\.Cited by:[§2\.3](https://arxiv.org/html/2611.21643#S2.SS3.p1.1)\.
- Redlineet al\.\(1995\)S\. Redline, P\. V\. Tishler, T\. D\. Tosteson, J\. Williamson, K\. Kump, I\. Browner, V\. Ferrette, and P\. KrejciThe familial aggregation of obstructive sleep apnea\.American journal of respiratory and critical care medicine151,pp\. 682–687\.Cited by:[§A\.7](https://arxiv.org/html/2611.21643#A1.SS7.p1.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p1.1)\.
- Schreineret al\.\(2023\)T\. Schreiner, M\. Petzka, T\. Staudigl, and B\. P\. StaresinaRespiration modulates sleep oscillations and memory reactivation in humans\.Nature Communications14\(1\),pp\. 8351\.Cited by:[§1](https://arxiv.org/html/2611.21643#S1.p1.1),[§2\.1](https://arxiv.org/html/2611.21643#S2.SS1.p1.1)\.
- Shuzanet al\.\(2024\)M\. N\. I\. Shuzan, M\. H\. Chowdhury, S\. B\. Alam, M\. B\. I\. Reaz, M\. S\. Khan, M\. Murugappan, and M\. E\. ChowdhuryPPG2RespNet: a deep learning model for respirational signal synthesis and monitoring from photoplethysmography \(ppg\) signal\.Physical and Engineering Sciences in Medicine47\(4\),pp\. 1705–1722\.Cited by:[§2\.1](https://arxiv.org/html/2611.21643#S2.SS1.p2.1)\.
- Slagle \(2024\)K\. SlagleSpacebyte: towards deleting tokenization from large language modeling\.Advances in Neural Information Processing Systems37,pp\. 124925–124950\.Cited by:[§2\.2](https://arxiv.org/html/2611.21643#S2.SS2.p1.1)\.
- Suet al\.\(2024\)J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. LiuRoformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§3\.4](https://arxiv.org/html/2611.21643#S3.SS4.p1.1)\.
- Thapaet al\.\(2026\)R\. Thapa, M\. R\. Kjaer, B\. He, I\. Covert, H\. Moore IV, U\. Hanif, G\. Ganjoo, M\. B\. Westover, P\. Jennum, A\. Brink\-Kjaer,et al\.A multimodal sleep foundation model for disease prediction\.Nature Medicine,pp\. 1–11\.Cited by:[§2\.1](https://arxiv.org/html/2611.21643#S2.SS1.p2.1)\.
- Van Den Oordet al\.\(2017\)A\. Van Den Oord O\. Vinyalset al\.Neural discrete representation learning\.Advances in neural information processing systems30\.Cited by:[§3\.3](https://arxiv.org/html/2611.21643#S3.SS3.p2.2)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[§3\.4](https://arxiv.org/html/2611.21643#S3.SS4.p1.1)\.
- Wanget al\.\(2018\)C\. Wang, J\. Zhang, and H\. ChenSemi\-autoregressive neural machine translation\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 479–488\.Cited by:[§2\.3](https://arxiv.org/html/2611.21643#S2.SS3.p1.1)\.
- Zhaet al\.\(2026\)K\. Zha, C\. Li, H\. He, P\. Cao, T\. Li, A\. Mirzazadeh, E\. Zhang, J\. W\. Lee, Y\. Kim, and D\. KatabiPhysiology as language: translating respiration to sleep eeg\.arXiv preprint arXiv:2602\.00526\.Cited by:[Appendix D](https://arxiv.org/html/2611.21643#A4.p1.1),[§1](https://arxiv.org/html/2611.21643#S1.p2.1),[§2\.1](https://arxiv.org/html/2611.21643#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p3.1)\.
- Zhanget al\.\(2018\)G\. Zhang, L\. Cui, R\. Mueller, S\. Tao, M\. Kim, M\. Rueschman, S\. Mariani, D\. Mobley, and S\. RedlineThe national sleep research resource: towards a sleep data commons\.Journal of the American Medical Informatics Association25\(10\),pp\. 1351–1358\.Cited by:[§A\.2](https://arxiv.org/html/2611.21643#A1.SS2.p1.1),[§A\.7](https://arxiv.org/html/2611.21643#A1.SS7.p1.1),[§4\.1](https://arxiv.org/html/2611.21643#S4.SS1.p1.1)\.
- Zhenget al\.\(2026\)R\. Zheng, W\. Liu, H\. Du, Q\. Zhang, C\. Deng, Q\. Chen, W\. Wang, Y\. Ai, and Z\. LingSay more with less: variable\-frame\-rate speech tokenization via adaptive clustering and implicit duration coding\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 35021–35029\.Cited by:[§2\.2](https://arxiv.org/html/2611.21643#S2.SS2.p2.1)\.
- Zhouet al\.\(2026\)H\. Zhou, S\. A\. Lee, C\. Tanade, K\. S\. Chun, J\. Lee, M\. Gwak, M\. Thukral, J\. Sung, E\. Hwang, M\. B\. Morshed, L\. Zhu, V\. Nathan, M\. M\. Rahman, S\. Venkatraman, and S\. A\. DesaiPhysiology\-aware masked cross\-modal reconstruction for biosignal representation learning\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=NxbSbkLNMc)Cited by:[§2\.1](https://arxiv.org/html/2611.21643#S2.SS1.p2.1)\.

## Appendix ADatasets

A brief overview of the datasets used in this study is provided below\. Table[5](https://arxiv.org/html/2611.21643#A1.T5)summarizes the main statistics of the datasets used for pretraining, while Table[6](https://arxiv.org/html/2611.21643#A1.T6)presents the datasets used for downstream evaluation\. Table[7](https://arxiv.org/html/2611.21643#A1.T7)reports the channels and sampling rates available for each dataset\.

Table 5:Statistics of training recordings \(minimum duration: 256 min\)\. Stage counts cover all annotated 30\-s epochs in retained recordings, whereas effective training duration includes only complete 256\-min input windows\.DatasetRecordsSubjectsAge \(years\)BMISex %\(Female/Male\)Stage countStage ratio \(%\)WN1N2N3REMTotalWN1N2N3REMPHY\-Train99099055\.2±\\pm14\.3N/A33 / 67145,317135,210371,718101,551113,765867,5611716421213PHY\-Test98998954\.8±\\pm14\.3N/A37 / 63N/AN/ASHHS13,6553,65563\.1±\\pm11\.528\.1±\\pm5\.252 / 48737,584136,1221,517,256471,476516,1693,378,607224451415SHHS22,5472,54767\.6±\\pm10\.328\.3±\\pm5\.054 / 46682,217106,8611,102,583302,150395,1812,588,992264431215KISS5,6755,67544\.6±\\pm14\.625\.6±\\pm4\.221 / 79946,226640,5321,516,640584,211648,1134,335,7222215351315KVSS80080051\.4±\\pm14\.526\.8±\\pm4\.424 / 76134,500122,688261,61846,39495,539660,739201940714MrOS12,7582,75876\.4±\\pm5\.527\.1±\\pm3\.80 / 100943,949129,0861,234,915221,090381,3662,910,40632443813MrOS299199181\.0±\\pm4\.426\.9±\\pm3\.80 / 100381,59780,258425,04544,896129,8531,061,64936840412MESA1631163169\.4±\\pm9\.128\.6±\\pm5\.553 / 47478,846165,620680,574117,046215,2911,657,377291041713

Table 6:The dataset statistics used for validation and test\.DatasetRecordsSubjectsAge \(years\)BMISex %\(Female/Male\)Stage countStage ratio \(%\)WN1N2N3REMTotalWN1N2N3REMSHHS1\-Val808061\.6±\\pm12\.028\.0±\\pm4\.354 / 4616,3962,91232,6369,31110,88272,137234451315SHHS1\-Test29029063\.9±\\pm11\.128\.4±\\pm4\.949 / 5157,93311,558122,03735,27442,105268,907224451316MESA\-Val17017069\.1±\\pm8\.628\.9±\\pm5\.459 / 4149,65716,26572,39812,46722,476173,26329942713MESA\-Test626268\.8±\\pm8\.929\.6±\\pm6\.052 / 4818,5696,28124,5705,4068,37363,199291039913CFS\-Train45245243\.1±\\pm19\.832\.0±\\pm9\.352 / 48101,44715,927175,73358,78959,978411,874254431415CFS\-Val505040\.4±\\pm19\.031\.4±\\pm8\.954 / 4612,3191,75119,7185,8796,09945,766274431313CFS\-Test10010038\.1±\\pm19\.331\.6±\\pm9\.062 / 3821,4563,04437,97413,12011,99787,591254431513

Table 7:Channels and sampling rates included in different modalities across datasets\. The manufacturer of PSG recording machine is also provided\. When sampling rates are same across different channels in the modality, we only write once at the top row\.ModalityKISS / KVSSSHHS1SHHS2PHYMESAMrOS1 / 2CFSCh\.SR \(Hz\)Ch\.SR \(Hz\)Ch\.SR \(Hz\)Ch\.SR \(Hz\)Ch\.SR \(Hz\)Ch\.SR \(Hz\)Ch\.SR \(Hz\)EEGC3\-M2200C3\-A2125C3\-A2125C3\-M2200C4\-M1256C3\-M2256C3\-A2128C4\-M1C4\-A1C4\-A1C4\-M1Oz\-CzC4\-M1C4\-A1O1\-M2O1\-M2Fz\-CzO1\-M2O2\-M1O2\-M1O2\-M1F3\-M2F4\-M1ECGECG200ECG125ECG125ECG250ECG256ECG1\-ECG2512ECG1\-ECG2256/512RespiratoryFlow200Flow10Flow10Flow200Flow32Flow64Nasal Pres64/128ThermistorThoraxThoraxThoraxThermistor16Airflow32ThoraxAbdomenAbdomenAbdomenThoraxThor Effort32AbdomenPressAbdomenAbdo Effort32PPG––––––––Pleth256––PlethWV128ManufacturerNox, EmblaCompumedicsCompumedicsUnknownCompumedicsCompumedicsCompumedics

### A\.1PhysioNet 2018 \(PHY\) Dataset

The PhysioNet 2018 dataset was originally developed for the 2018 PhysioNet/CinC Challenge[Goldberger et al\. \(2000\)](https://arxiv.org/html/2611.21643#bib.bib19);[Ghassemi et al\. \(2018\)](https://arxiv.org/html/2611.21643#bib.bib20)\. It was provided by the Computational Clinical Neurophysiology Laboratory and the Clinical Data Animation Laboratory at Massachusetts General Hospital\. Although the main objective of the challenge was arousal detection, the dataset also contains expert\-annotated sleep stages for 994 subjects\. In addition, 991 recordings were reserved as a test set, although the corresponding labels remain private\. In this study, both the labeled training set and the unlabeled test set were used for trainingSleep2Word\. The original PhysioNet recordings were acquired at dataset\-specific sampling rates, with most physiological signals sampled at 200 Hz\. During preprocessing, the raw biosignals were resampled to 100 Hz to ensure a consistent sampling rate across datasets\. Sleep stages were scored following the American Academy of Sleep Medicine \(AASM\) guidelines\.

### A\.2Sleep Heart Health Study \(SHHS\) Dataset

The Sleep Heart Health Study \(SHHS\)[Zhang et al\. \(2018\)](https://arxiv.org/html/2611.21643#bib.bib21);[Quan et al\. \(1997\)](https://arxiv.org/html/2611.21643#bib.bib22)is a multicenter cohort study organized by the National Heart, Lung, and Blood Institute\. The dataset contains data collected during two separate visits\. SHHS1 corresponds to the first visit, conducted between 1995 and 1998, and included 6,441 men and women aged 40 and older\. SHHS2 corresponds to the second visit, carried out between 2001 and 2003, and included 3,295 participants\. Polysomnography \(PSG\) recordings were collected at home by trained technicians and included several physiological signals, such as EEG \(C3\-A2 and C4\-A1\), two EOG channels, EMG, respiratory effort, airflow, oxygen saturation, ECG, and body position\.

For our experiments, the SHHS1 dataset was divided into 3,655 subjects for training, 80 subjects for validation, and 290 subjects for the final held\-out test set\. For survival prediction, nine training subjects were excluded due to unavailable features or zero follow\-up time, and one test subject with zero follow\-up time was excluded, yielding 3,646 training, 80 validation, and 289 test subjects for analysis\. The test set was restricted to subjects belonging to the common evaluation set shared between our pipeline and the wav2Sleep baseline, allowing a consistent comparison between the two models\. Subjects from the original held\-out test set that were not part of this common subset were kept unused\. The SHHS2 dataset was used for pretraining and was not used for downstream tasks\.

### A\.3Korea Image\-based Sleep Study \(KISS\) Dataset

The Korea Image\-based Sleep Study \(KISS\) dataset[Jeong et al\. \(2023\)](https://arxiv.org/html/2611.21643#bib.bib28)is a standardized polysomnography \(PSG\) dataset collected between 2013 and 2020 across four sleep centers\. Recordings were acquired using Embla and NOX\-A1 PSG systems, resulting in a total of 10,253 recordings\. Sleep stages were scored by experts according to the AASM version 2\.6 guidelines[Berry \(2012\)](https://arxiv.org/html/2611.21643#bib.bib1);[Berry et al\. \(2017\)](https://arxiv.org/html/2611.21643#bib.bib2)\. Each recording contains 21 different biosignals, including several EEG, EOG, and EMG channels, as well as respiratory and movement\-related signals\. The dataset is publicly available through AI Hub[AIHub \(2020\)](https://arxiv.org/html/2611.21643#bib.bib26)\.

The training split defined in[Jeong et al\. \(2023\)](https://arxiv.org/html/2611.21643#bib.bib28)are used for training\. KISS was not used for downstream evaluation or final testing\.

### A\.4Korea Video Sleep Study \(KVSS\) Dataset

The Korea Video Sleep Study \(KVSS\) dataset is a retrospectively constructed multicenter clinical cohort that includes synchronized infrared sleep video and polysomnography \(PSG\) recordings with expert annotations[Choi et al\. \(2026\)](https://arxiv.org/html/2611.21643#bib.bib27)\. Data were collected under IRB\-approved protocols at three hospitals: Chungnam National University Hospital, The Catholic University of Korea St\. Vincent’s Hospital, and Hallym University Hospital\. The complete KVSS cohort contains 936 PSG examinations with synchronized infrared video, while 881 examinations were retained in this study after screening and quality control\. The recordings were obtained during routine clinical sleep studies for various sleep\-related conditions, including suspected obstructive sleep apnea, insomnia, periodic limb movement disorder, and REM sleep behavior disorder\. Infrared video was recorded simultaneously with PSG to maintain temporal synchronization\. Although camera position and lighting conditions differed slightly between sites, all videos were standardized to MP4 format with a resolution of640×480640\\times 480pixels at 5 fps\. PSG recordings were stored in European Data Format \(EDF\)\. Sleep stages \(Wake, N1, N2, N3, and REM\) and AASM\-defined events were annotated by certified sleep technologists and reviewed by sleep physicians\. The dataset was de\-identified, and expert cross\-checking was performed to verify synchronization and correct obvious scoring inconsistencies\. In our experiments, KVSS was used as a PSG dataset for pretraining\. Only the physiological signals required by the model were processed, while the synchronized infrared video data were not used\.

### A\.5Multi\-Ethnic Study of Atherosclerosis \(MESA\) Dataset

The Multi\-Ethnic Study of Atherosclerosis \(MESA\)[Chen et al\. \(2015\)](https://arxiv.org/html/2611.21643#bib.bib23)is a multicenter cohort including 6,814 adults aged between 45 and 84 years from four racial and ethnic groups \(White, Black, Hispanic, and Chinese\-American participants\)\. As part of Exam 5, conducted between 2010 and 2013, the MESA Sleep study enrolled 2,237 participants who completed a single night of unattended in\-home polysomnography \(PSG\) together with wrist actigraphy\. PSG recordings were set up during evening home visits by trained staff, while sleep stages and respiratory events were scored at a centralized sleep reading center using standardized procedures\. The National Sleep Research Resource \(NSRR\) provides the PSG recordings in European Data Format \(EDF\), together with XML annotation files containing sleep\-stage and respiratory\-event information\. Respiratory\-event annotations were harmonized through rule\-based post\-processing of the original labels to ensure consistent criteria across datasets, following the procedure described in[Ahn et al\. \(2026\)](https://arxiv.org/html/2611.21643#bib.bib25)\. The PSG recordings include EEG \(Fz\-Cz, Cz\-Oz, and C4\-A1\), EOG, EMG, ECG, nasal airflow, thoracic and abdominal respiratory effort, oxygen saturation, and body position\. In our experiments, the original MESA training split was kept unchanged with 1,631 recordings\. To ensure a consistent comparison with the wav2sleep baseline, the validation set was filtered to 170 recordings, while the final held\-out test set was restricted to 62 recordings shared between the two evaluation pipelines\. Recordings from the original test set that were not part of this common subset were kept unused\.

### A\.6Osteoporotic Fractures in Men Study \(MrOS\) Dataset

The Osteoporotic Fractures in Men Sleep Study \(MrOS Sleep\)[Blackwell et al\. \(2011\)](https://arxiv.org/html/2611.21643#bib.bib24)is a multicenter cohort of older men aged 65 years and above\. It includes unattended in\-home polysomnography \(PSG\), which was set up by trained technicians and annotated using standardized scoring procedures\. The study was conducted between December 2003 and March 2005, during which 3,135 participants from the original MrOS cohort of 5,994 men completed overnight unattended PSG\. The dataset available through the National Sleep Research Resource \(NSRR\) provides PSG recordings in European Data Format \(EDF\), together with XML annotation files\. These recordings contain multiple physiological signals, including EEG, EOG, EMG, ECG, nasal airflow, thoracic and abdominal respiratory effort, oxygen saturation, and body position\. Respiratory\-event annotations were harmonized using rule\-based post\-processing of the original labels to maintain consistent criteria across datasets, following the procedure described in[Ahn et al\. \(2026\)](https://arxiv.org/html/2611.21643#bib.bib25)\. In our experiments, data from both MrOS visits were included in the pretraining dataset\. For MrOS1, the C3\-A2 EEG channel and abdominal respiratory effort were used, resulting in 2,758 recordings\. For MrOS2, the C3\-M2 EEG channel and abdominal respiratory effort were used, resulting in 991 recordings\. A subset of MrOS2 recordings exhibited a different signal amplitude scale and was harmonized during preprocessing\.

### A\.7Cleveland Family Study \(CFS\) Dataset

The Cleveland Family Study \(CFS\)\([Zhang et al\., 2018](https://arxiv.org/html/2611.21643#bib.bib21);[Redline et al\., 1995](https://arxiv.org/html/2611.21643#bib.bib29)\)is a family\-based cohort designed to investigate familial risk factors associated with sleep apnea\. The cohort includes 2,284 participants from 361 families who were followed across several visits over a period of approximately 16 years\. The National Sleep Research Resource \(NSRR\) provides overnight polysomnography recordings, including full\-night PSG data collected during Visit 5\. Subjects younger than 14 years were excluded because pediatric sleep studies follow different clinical scoring criteria\. The available PSG signals and corresponding sleep\-stage annotations were processed using the same preprocessing pipeline applied to the other datasets\.

CFS is excluded fromSleep2Word’s training stages\. For downstream evaluation, however, we train a separate task\-specific head using labels from the CFS training split and evaluate it on the held\-out CFS test split\. Thus, the CFS results assess the transfer of a fixed translation model to an unseen cohort under cohort\-specific supervised downstream evaluation, rather than zero\-shot transfer of the complete prediction pipeline\. In contrast, CFS is an internal dataset for the wav2sleep baseline, as it is included among the datasets used for model training and evaluation\. Therefore, while the wav2sleep results are computed on its held\-out CFS test split, CFS does not represent an unseen dataset for wav2sleep, unlike in our evaluation setting\.

## Appendix BModel Specification

Table[8](https://arxiv.org/html/2611.21643#A3.T8)and Table[9](https://arxiv.org/html/2611.21643#A3.T9)summarizeSleep2Wordand CNN\-Transformer model \(C\-T baseline\) specifications, respectively\. C\-T baseline is trained with the same training recipe as translation model, described in Table[11](https://arxiv.org/html/2611.21643#A3.T11)\.

We also evaluated larger C\-T baseline variants with increasing model capacity and window size\. These variants did not converge reliably under the varied training configurations\. One of the tested larger model specification and training results are provided in Table[10](https://arxiv.org/html/2611.21643#A3.T10)\. This observation is limited to the settings tested and does not rule out improvements from further optimization\.

## Appendix CImplementation Details

We provide training hyperparameters in Table[11](https://arxiv.org/html/2611.21643#A3.T11)\. The hyperparameters used for each downstream tasks are summarized in Table[12](https://arxiv.org/html/2611.21643#A3.T12)\. We have trained and evaluatedSleep2Wordand baseline methods on a single NVIDIA H200 GPU \(140 GiB HBM3e\) with two Intel Xeon Platinum 8462Y\+ CPUs\. Models are implemented in PyTorch 2\.5\.1 \(CUDA 12\.1, cuDNN 9\.1\) with PyTorch Lightning 2\.6\.0, and trained in bf16 mixed precision with distributed data parallelism\. End\-to\-end training takes approximately 10 hours per single modality\.

PPG pretraining\. PPG channel is not included in SleepMaMi configuration and is only available in MESA dataset\. To tackle this, we adopt two\-stage pretraining strategy\. First, we pretrain SleepMaMi using three modalities \(EEG, ECG and Respiratory\) on the full training dataset\. We then introduce a PPG\-specific private encoder and fine\-tune only this encoder on the MESA PPG data while keeping all other modules frozen\. This procedure maps PPG inputs into the common representation space learned from the other modalities, enabling alignment across all four modalities\.

Sleep\-stage classification\.We use a CNN\-TCN architecture\. CNN layers extract local spectral patterns from the reconstructed EEG spectrogram\. The TCN then models temporal dependencies across consecutive sleep epochs using dilated convolutions\. The final features are projected to four sleep\-stage classes\. The model is trained for 20 epochs with weighted cross\-entropy loss\.

Wordbook initialization\. As described in Section[3](https://arxiv.org/html/2611.21643#S3), the wordbook is initialized using data\-driven approach\. Wordization is disabled for the firstIIbatches, during which chunk embeddings and their lengths are collected\. The dominant chunk lengths are then identified, and chunks are allocated to these length groups in proportion to the number of chunks\. Within each group, the collected chunk embeddings are clustered once using Lloyd’s algorithm to initialize the corresponding*words*\.

α\\alphais calibrated as the median distance between each chunk to the nearest centroid of a different length group, divided by the smallest log\-duration gap between initial length groups\. A smaller value may underweight duration, causing chunks with similar contents but different lengths to be assigned to the same word\. In contrast, a largeα\\alphamay cause the duration to dominate content, making assignment primarily length\-dependent\. This calibration balances the contributions of content and duration to word assignment\.

PasL reproduction\. We reproduce PasL using the architecture and training setup described in the original paper\. EEG spectrograms have a resolution of256×512256\\times 512\. A VQGAN tokenizer maps each spectrogram to an8×648\\times 64grid of 512 discrete tokens using an 8,192\-entry codebook\. The masked Transformer uses an 8\-layer encoder and decoder, 768\-dimensional hidden states, and 8 attention heads\. The model is trained for 500 epochs with AdamW using the masked\-token cross\-entropy \(CE\) objective\. Reconstructed EEG spectrograms are obtained with the frozen VQGAN decoder\. Training converges stably, although our reproduced downstream performance remains slightly below the values reported in the original paper\.

Table 8:Sleep2Wordmodel specifications\.DDdenotes the shared latent width; one base token covers 1 s of signal \(30 tokens per epoch\),L′L^\{\\prime\}is the number of base tokens andCmC^\{m\}is the number of words in a full night recording\.ComponentModuleConfigurationOutputNotesFrozen Base EncoderSleepMaMiMicro\-encoder private and shared modules\(L′,256\)\(L^\{\\prime\},256\)Parameters≈\\approx20\.4M totalChunkingChunk encoder2×2\\timesMamba blocks\(L′,256\)\(L^\{\\prime\},256\)\(dstate=16d\_\{\\mathrm\{state\}\}\{=\}16, conv=4, expand=2, Gated MLP with×4\\times 4expansion\)RouterWq,Wk∈ℝ256×256W\_\{q\},W\_\{k\}\\in\\mathbb\{R\}^\{256\\times 256\}\(L′,256\)\(L^\{\\prime\},256\)EncoderEncoder4×4\\times\(2 sliding\-window Transformer \+ 1 Full Transformer\)\(Cm,256\)\(C^\{m\},256\)D=256D\{=\}256, 8 heads, Gated MLP with×4\\times 4expansion, RoPE at word startDecoderBlock ARDepth 2, Self \+ Cross attention per sync block\(CE​E​G,\|𝒲\|\+1\)\(C^\{EEG\},\|\\mathcal\{W\}\|\+1\)id head, length head\(CE​E​G,2​r\+1\)\(C^\{EEG\},2r\+1\)rr: Residual slot sizeDechunkingWordbook lookupScatter to the base grid \(STE\)\(T,256\)\(T,256\)Dechunk encoder2×2\\timesMamba blocks\(T,256\)\(T,256\)Reconstruction headSpectrogram headConv1d\(\(k,s\)=\(30,6\)\(k,s\)\{=\}\(30,6\)\)→\\toConv1d\(\(k,s\)=\(5,5\)\(k,s\)\{=\}\(5,5\)\)\(Ty,NfT\_\{y\},N\_\{f\}\)Only for EEG→\\toLinear \(256→Nf256\{\\to\}N\_\{f\}\)Upsampling headConv1d\(\(k,s\)=\(30,1\)\(k,s\)\{=\}\(30,1\)\)→\\toConv1d\(\(k,s\)=\(30,1\)\(k,s\)\{=\}\(30,1\)\)\(L,1\)\(L,1\)for peripherals→\\toLinear \(256→fs256\{\\to\}f\_\{s\}\)\(L=L′×Nf\)\(L=L^\{\\prime\}\\times N\_\{f\}\)Parameters≈\\approx25\.8M total

Table 9:CNN\-Transformer baseline specifications\.LLis the total length of full\-night vector \(100 Hz×\\timesthe number of seconds in full\-night recording\)Table 10:Larger CNN\-Transformer baseline specifications and training results\.Table 11:Hyperparameter settings for training\.Table 12:Training details used for downstream tasks\.
## Appendix DEEG Spectrogram

InSleep2Word, we model the target EEG as a time\-frequency representation rather than as the raw waveform\. Although peripheral signals encode information coupled to brain activity, they are unlikely to support reliable recovery of the fine\-grained, sample\-level details of EEG waveforms\. In contrast, the EEG spectrogram provides a physiologically meaningful and temporally localized representation of spectral activity that is more suitable for cross\-modal estimation, as suggested by[Zha et al\. \(2026\)](https://arxiv.org/html/2611.21643#bib.bib12)\.

The EEG spectrogram is built in the following procedure\. Each EEG recording is normalized by the standard deviation of each dataset’s training split and partitioned into non\-overlapping windows ofω\\omegaseconds, from which log\-power multi\-taper spectra\([Prerau et al\., 2017](https://arxiv.org/html/2611.21643#bib.bib30)\)are computed over 0\.5\-32 Hz\. Any windows carrying no usable signal \(flat or non\-finite, e\.g\., a detached electrode\) are set to zero\. The dB values are subsequently mapped to \[0,1\] using the 0\.1st and 99\.9th percentiles measured over the training corpus\. The same percentile\-based mapping is applied across datasets and splits after dataset\-specific amplitude normalization\. Accordingly, the resulting values represent power on a common normalized scale\. We useω\\omegaof five seconds as our default setting, except the reconstruction experiment where thirty seconds is used for fair comparison to other baselines\.

## Appendix EModifications to SleepMaMi architecture

We use the shared embeddings from SleepMaMi\([Park et al\., 2026](https://arxiv.org/html/2611.21643#bib.bib11)\), with several architectural modifications to support translation from a single source modality\. In the original architecture, the shared module receives the concatenated embeddings of all available modalities\. Consequently, generating shared embeddings requires access to multiple modalities at once, including the target modality \- this is not permissible in our translation setting\. We therefore modify the shared module to process each modality independently and align the resulting representations across modalities\. This design allows the model to encode a source signal without requiring the target signal while preserving a common cross\-modal representation space\.

We also modify the original leave\-one\-out contrastive learning \(LOO\-CL\) objective\. For each source modality, we randomly sample a subset of the remaining modalities and contrast the representations averaged over them with that of the source modality\. Exposing the model to different modality subsets encourages it to capture physiological features that are consistent across diverse combinations of modalities\.

Finally, we modify the fusion module to combine the shared embeddings obtained from a randomly selected subset of modalities with the private embeddings\. This modification further encourages the shared representations to encode modality\-agnostic information\.

![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/mami.jpg)\(a\)Shared module modification
![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/loo_cl_comparison.jpg)\(b\)LOO\-CL objective modification

Figure 5:Overview of the modifications to the SleepMaMi architecture\. \(a\) The shared module processes each modality independently, enabling single\-modality inputs while preserving a common representation space\. \(b\) LOO\-CL is modified to handle different subsets of available modalities\.
## Appendix FAuxiliary losses

This section provides a more detailed description of auxiliary losses used during the context\-aware chunking stage\.

Ratio loss \(ℒratio\\mathcal\{L\}\_\{\\mathrm\{ratio\}\}\): We adopt this loss to regulate the proportion of predicted boundaries\. The formulation follows[Hwang et al\. \(2026\)](https://arxiv.org/html/2611.21643#bib.bib18), whereNNcontrols the target chunking ratio\.

ℒr​a​t​i​o\\displaystyle\\mathcal\{L\}\_\{ratio\}=NN−1​\(\(N−1\)​F​G\+\(1−F\)​\(1−G\)\),F=1L′​∑i𝟙pi≥0\.5,G=1L′​∑ipi\\displaystyle=\\frac\{N\}\{N\-1\}\(\(N\-1\)FG\+\(1\-F\)\(1\-G\)\),\\quad F=\\frac\{1\}\{L^\{\\prime\}\}\\sum\_\{i\}\\mathbbm\{1\}\_\{p\_\{i\}\\geq 0\.5\},\\quad G=\\frac\{1\}\{L^\{\\prime\}\}\\sum\_\{i\}p\_\{i\}\(9\)
Alignment loss \(ℒalign\\mathcal\{L\}\_\{\\mathrm\{align\}\}\): This loss prevents the refined embeddings of chunk encoder from deviating substantially to modality\-specific representation space\.

ℒa​l​i​g​n\\displaystyle\\mathcal\{L\}\_\{align\}=∑c=1C‖𝐳¯c−𝐞¯c‖22,𝐞¯c=1lc​∑t=scsc\+lc−1𝐞t\\displaystyle=\\sum\_\{c=1\}^\{C\}\\\|\\bar\{\\mathbf\{z\}\}\_\{c\}\-\\bar\{\\mathbf\{e\}\}\_\{c\}\\\|\_\{2\}^\{2\},\\quad\\bar\{\\mathbf\{e\}\}\_\{c\}=\\frac\{1\}\{l\_\{c\}\}\\sum\_\{t=s\_\{c\}\}^\{s\_\{c\}\+l\_\{c\}\-1\}\\mathbf\{e\}\_\{t\}\(10\)
Barrier loss \(ℒbarrier\\mathcal\{L\}\_\{\\mathrm\{barrier\}\}\): This loss encourages boundaries to be inserted at regular temporal intervals, thereby the resulting chunks are well placed for the block autoregressive decoder\.

ℒbarrier=−∑a∈𝒜log\(pa\),where𝒜=\{1\+jB:j≥1,1\+jB≤L′\}\\mathcal\{L\}\_\{\\mathrm\{barrier\}\}=\-\\sum\_\{a\\in\\mathcal\{A\}\}\\text\{log\}\(p\_\{a\}\),\\quad\\text\{where\}\\quad\\mathcal\{A\}=\\\{1\+jB:j\\geq 1,1\+jB\\leq L^\{\\prime\}\\\}\(11\)

## Appendix GAblation

We conduct ablation study to investigate the individual contribution of our core components\.

Loss components\. Table[13](https://arxiv.org/html/2611.21643#A7.T13)provides the ablation results for the auxiliary loss components\. We conduct this experiment using the SHHS1 training set and evaluate respiratory\-to\-EEG sleep staging accuracy on SHHS1 under the linear\-probing protocol\. RemovingℒBarrier\\mathcal\{L\}\_\{\\mathrm\{Barrier\}\}results in accuracy decrease of 0\.007 \(0\.7 percentage points\), demonstrating its contribution to maintaining temporal alignment between the source and target sequences\. RemovingℒAlign\\mathcal\{L\}\_\{\\mathrm\{Align\}\}andℒBarrier\\mathcal\{L\}\_\{\\mathrm\{Barrier\}\}decreases accuracy by 0\.104 \(10\.4 percentage points\), suggesting that the alignment objective prevents the refined chunk representations from drifting toward modality\-specific features\. We do not report the configuration withoutℒRatio\\mathcal\{L\}\_\{\\mathrm\{Ratio\}\}, since removing it causes the chunking mechanism to collapse and does not provide meaningful results\.

Table 13:Ablation on loss components\.Chunking mechanism\. To assess the effectiveness of*context\-aware chunking*and*wordization*, we compareSleep2Wordagainst two alternative methods:*Base2base*performs no chunking or wordization and directly translates the source base embeddings𝐞m\\mathbf\{e\}^\{m\}into the target base embeddings𝐞E​E​G\\mathbf\{e\}^\{EEG\}\.*FixedPool*replaces context\-aware chunking with average pooling over non\-overlapping windows of four base embeddings, matching the chunking ratio ofN=4N=4\. Table[15](https://arxiv.org/html/2611.21643#A7.T15)provides respiratory\-to\-EEG sleep staging accuracy on SHHS1 under the linear\-probing protocol, using SHHS1 training set\.Sleep2Wordoutperforms Base2base by 0\.022 \(2\.2 percentage points\), demonstrating that word\-level abstraction improves downstream performance\. It also surpasses FixedPool by 0\.020 \(2\.0 percentage points\), showing that adaptively locating chunk boundaries preserves more task\-relevant temporal structure at a comparable compression level\. These gains highlight that the benefit ofSleep2Wordarises not only from shorter sequences and improved computational efficiency, but also from learning context\-aware, physiologically meaningful abstractions\.

Duration awareness and duration residual prediction\. Each word inSleep2Wordconsists of a content componentνk∈ℝd\\nu\_\{k\}\\in\\mathbb\{R\}^\{d\}and a duration componentηk∈ℝ\\eta\_\{k\}\\in\\mathbb\{R\}\. The decoder additionally predicts an occurrence\-specific duration residual to account for variations among chunks assigned to the same word\. Table[15](https://arxiv.org/html/2611.21643#A7.T15)evaluates the contributions of these two components\. We conduct this experiment using the SHHS1 training set and report respiratory\-to\-EEG sleep staging accuracy on SHHS1 under the linear\-probing protocol\.

In the content\-only variant, the word identity is determined solely by the content\. Incorporating the duration termηk\\eta\_\{k\}in the wordbook enhances accuracy from 0\.732 to 0\.748, an increase of 0\.016 \(1\.6 percentage points\)\. This result suggests that explicitly distinguishing chunks with similar content but different durations improves temporal alignment during translation\. Residual prediction further improves accuracy by 0\.005 \(0\.5 percentage points\), indicating that modeling deviations from the prototypical duration helps reduce residual temporal discrepancy\.

Table 14:Ablation on chunking mechanism\.
Table 15:Ablation on duration modeling\.

Wordbook size\. Table[18](https://arxiv.org/html/2611.21643#A7.T18)provides the effect of varyingKK\. We conduct this experiment using the full training set and evaluate respiratory\-to\-EEG sleep staging accuracy on SHHS1 under the linear\-probing protocol, while fixingB=3B=3andN=4N=4\. Performance improves as the wordbook size increases, with the gains saturating atK=2048K=2048\. This suggests that a sufficiently large vocabulary is required to capture diverse patterns, while further increasing its size provides limited benefit\.

Target chunking ratio\. Table[18](https://arxiv.org/html/2611.21643#A7.T18)provides the results for different values ofNN\. We conduct this experiment using the full training set and evaluate respiratory\-to\-EEG sleep staging accuracy on SHHS1 under the linear\-probing protocol, while fixingK=2048K=2048andB=3B=3\. Increasing the chunking ratio up toN=4N=4does not harm the model performance, but larger values lead to a modest decrease in accuracy\. These results suggest that the wordization mechanism can effectively compress the target sequence while preserving task\-relevant information\. The experiment is done under linear probing protocol using the full training set and respiratory\-to\-EEG sleep staging accuracy on SHHS is reported\.

Block size\. Table[18](https://arxiv.org/html/2611.21643#A7.T18)provides the effect of varyingBB\. We conduct this experiment using the full training set and evaluate respiratory\-to\-EEG sleep staging accuracy on SHHS1 under the linear\-probing protocol, while fixingK=2048K=2048andN=4N=4\. Accuracy decreases as the block size increases, potentially because temporal misalignment accumulates over longer generation spans\. This result highlights the importance of introducing block boundaries at appropriate intervals to limit temporal drift\.

Table 16:Ablation onKK\.
Table 17:Ablation onNN\.
Table 18:Ablation onBB

## Appendix HCompression Ratio

The modality\-specific wordbooks reduce sequence length by representing each variable\-length chunk with a single token\. We quantify this effect using pretrained checkpoints and 50 subjects from SHHS1 or MESA \(PPG\)\. Specifically, we define the effective compression ratio as100​\(1−C/L′\)100\(1\-C/L^\{\\prime\}\), whereL′L^\{\\prime\}andCCdenote the number of base patches and resulting word tokens, respectively\. All measurements are obtained withN=4N=4\.

As presented in Table[19](https://arxiv.org/html/2611.21643#A8.T19), the wordbooks reduce the full\-night sequence length by 64\.3% to 69\.7%, depending on the modality\. These empirical reductions are moderately below the target of 75%, implied byN=4N=4\. This discrepancy arises because chunk boundaries and lengths are adaptively determined based on their context\.

Table 19:Effective compression ratio\.
## Appendix IGenerated Spectrogram Samples\.

We provide selected samples of generated spectrograms using respiratory, ECG and PPG, respectively in Figure[6](https://arxiv.org/html/2611.21643#A9.F6),[7](https://arxiv.org/html/2611.21643#A9.F7), and[8](https://arxiv.org/html/2611.21643#A9.F8)\.

![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/spec_RESP_kiss_D2019-NX-01-0063_F64.png)\(a\)Spectrogram : 64\-year\-old Female
![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/spec_RESP_shhs1_shhs1-205267_F65.png)\(b\)Spectrogram : 65\-year\-old Female

Figure 6:Generated spectrograms from respiratory\.![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/spec_ECG_kiss_A2015-EM-01-0091_M46.png)\(a\)Spectrogram : 46\-year\-old Male
![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/spec_ECG_shhs1_shhs1-202544_F79.png)\(b\)Spectrogram : 79\-year\-old Female

Figure 7:Generated spectrograms from ECG\.![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/spec_PPG_cfs_cfs-visit5-800340_M55.png)\(a\)Spectrogram : 55\-year\-old Male
![Refer to caption](https://arxiv.org/html/2611.21643v1/figures/spec_PPG_mesa_mesa-sleep-2952_M65.png)\(b\)Spectrogram : 65\-year\-old Male

Figure 8:Generated spectrograms from PPG\.

相似文章

非侵入式EEG解码默读

Hacker News Top

这项研究使用非侵入式EEG和对比学习来解码默读过程中的单词,展示了可扩展的词汇信息恢复,其性能随数据量增加而提升。

MEL:用于fMRI翻译的坐标保持型EEG标记化

arXiv cs.LG

本文介绍了MEL,一个用于EEG到fMRI翻译的坐标保持型EEG标记化框架,通过显式建模血流动力学延迟和频谱空间动态,解决表示接口不匹配问题,并改善预测性能优于基线方法。