The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning

# The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning
Source: [https://arxiv.org/html/2608.12695](https://arxiv.org/html/2608.12695)
Ahmed SamehAffiliation:Computer Science & Engineering, University of Minnesota Twin Cities, Minneapolis, MN 55455\.\{sameh002, yvaratha\}@umn\.eduRamzi Al\-SharawiAffiliation:Robotics, University of Minnesota Twin Cities, Minneapolis, MN 55455\.\{alsha192\}@umn\.eduYogatheesan VaratharajahAffiliation:Computer Science & Engineering, University of Minnesota Twin Cities, Minneapolis, MN 55455\.\{sameh002, yvaratha\}@umn\.edu

###### Abstract

Self\-supervised electrocardiogram \(ECG\) models are often trained on a few seconds of ECG signal and, increasingly, on discretized token sequences\. It remains unclear whether these choices sacrifice information needed for rhythm inference and longitudinal consistency in real\-world ambulatory recordings\. We present a controlled study on the Icentia11k single\-lead dataset that varies \(i\) the input horizon \(16 seconds, 1 minute, 5 minutes, and 10 minutes\) and \(ii\) the front\-end representation \(continuous convolutional patch embeddings vs\. fixed vector\-quantized tokens\), while holding the Transformer backbone and training protocol constant\. Representations are assessed by downstream abnormal rhythm detection and by patient\-level retrieval that probes cross\-session stability\. Our results show that increasing temporal context beyond 16\-second snapshots yields stronger transfer and higher retrieval accuracy, with the strongest performance achieved by the 5\- and 10\-minute models, indicating improved capture of slow\-varying rhythm dynamics and individual\-specific structure\. Across all evaluated horizons, continuous patch embeddings outperform discretized tokens, suggesting that quantization can discard clinically relevant waveform detail\. These findings motivate ECG foundation models that emphasize extended context and continuous encoders for clinical prediction and similarity\-based applications\. Our code and pretrained models are publicly available at[https://github\.com/muha\-0/ecg\-ssl\-representation\-learning](https://github.com/muha-0/ecg-ssl-representation-learning)\.

###### Index Terms:

electrocardiography, self\-supervised learning, foundation models, temporal context, representation learning

## IIntroduction

Electrocardiography \(ECG\) is a fundamental tool for cardiovascular assessment, capturing cardiac electrophysiological activity via a non\-invasive, inexpensive measurement\. Recent advances in deep learning have demonstrated strong performance on ECG\-based diagnostic tasks, often approaching expert\-level interpretation\[[1](https://arxiv.org/html/2608.12695#bib.bib1),[2](https://arxiv.org/html/2608.12695#bib.bib2),[6](https://arxiv.org/html/2608.12695#bib.bib5)\]\. However, the success of these supervised approaches is tethered to large, carefully curated labeled datasets that are costly to obtain and difficult to scale across diverse patient populations\. Consequently, self\-supervised learning \(SSL\) has emerged as a leading paradigm for ECG representation learning, enabling models to exploit abundant unlabeled recordings and transfer learned features to clinical tasks\[[5](https://arxiv.org/html/2608.12695#bib.bib4),[12](https://arxiv.org/html/2608.12695#bib.bib7)\]\. Recently, this has culminated in the development of large\-scale foundation models designed for scalability and transferability\[[11](https://arxiv.org/html/2608.12695#bib.bib8),[10](https://arxiv.org/html/2608.12695#bib.bib9)\]\. Two design trends have become increasingly common in these models: short\-context training windows and discretized tokenization\.

Most existing SSL frameworks prioritize computational efficiency by training on temporal windows of only a few seconds\[[13](https://arxiv.org/html/2608.12695#bib.bib10),[4](https://arxiv.org/html/2608.12695#bib.bib11),[12](https://arxiv.org/html/2608.12695#bib.bib7)\]\. While short windows effectively capture local morphology, they provide insufficient rhythm context to represent physiological patterns that unfold over minutes, including baseline morphology, rhythm variability, and patient\-specific characteristics\. In parallel, inspired by natural language processing, several recent approaches discretize continuous ECG signals into symbolic tokens\[[16](https://arxiv.org/html/2608.12695#bib.bib12),[7](https://arxiv.org/html/2608.12695#bib.bib13)\]\. While tokenization can enable compact representations and sequence modeling, discretization imposes an information bottleneck that may obscure subtle morphological cues relevant to downstream clinical performance and patient\-level structure\.

In this work, we hypothesize that longer temporal context and continuous encoders are essential for learning physiologically coherent and patient\-consistent ECG representations\. We posit that extending the temporal context enables models to integrate information across multiple beats and rhythm states, improving the modeling of both transient and sustained cardiac phenomena, while avoiding discretization preserves fine\-grained morphological detail that may be important for downstream clinical and patient\-level tasks\. We test this hypothesis by comparing self\-supervised models trained across four temporal windows, using both continuous 1D convolutional neural network \(CNN\) and discretized vector quantization \(VQ\) encoders under a unified experimental framework\.

We perform evaluations along two complementary axes using the publicly available Icentia11K dataset\[[17](https://arxiv.org/html/2608.12695#bib.bib3)\]\. First, we assess clinical utility through a downstream atrial fibrillation \(AFib\) and atrial flutter \(AFL\) classification task\. Second, we quantify patient\-specific structure using Recall@k retrieval metrics to measure the consistency of patient groupings across time and activity\. This patient\-level evaluation is motivated by prior work emphasizing patient\-consistent representation learning and by the potential for embeddings to support similarity\-based applications such as cohort stratification and subgroup discovery\[[8](https://arxiv.org/html/2608.12695#bib.bib6)\]\. Our results indicate the following:

- •Moving beyond short ECG snapshots improves downstream classification and patient\-level consistency, with the largest gains observed at 5 and 10 minutes, suggesting that longer horizons are essential for integrating slow\-varying rhythm dynamics and long\-term signatures\.
- •Across all tested scales, continuous CNN patch embeddings consistently outperform VQ\-based tokenization, indicating that discretization introduces a quantization bottleneck that erodes high\-resolution waveform characteristics critical for both diagnostic accuracy and patient\-specific structure\.

## IIRelated Work

Early work in ECG\-SSL leveraged contrastive objectives to reduce dependence on labeled data\. CLOCS introduced patient\-consistent positive pairs, arguing that representations from the same individual should remain invariant across temporal segments and leads\[[8](https://arxiv.org/html/2608.12695#bib.bib6)\]\. CLOCS also noted that this shared context may diminish when recordings are separated by long time spans or reflect different activity states, a concern that is especially relevant for ambulatory datasets such as Icentia11k\[[17](https://arxiv.org/html/2608.12695#bib.bib3)\]\. Mehari and Strodthoff provided a comprehensive benchmark of self\-supervised learning on clinical 12\-lead ECGs and showed that an adaptation of contrastive predictive coding achieves linear\-evaluation performance within about 0\.5 percentage points of a supervised counterpart, establishing a competitive short\-window baseline\[[12](https://arxiv.org/html/2608.12695#bib.bib7)\]\.

Despite the shift toward large\-scale foundation models, current pipelines remain mostly snapshot\-focused\. Recent architectures like ST\-MEM utilize masked auto\-encoding of segments of only 10 seconds\[[13](https://arxiv.org/html/2608.12695#bib.bib10)\]\. Even as scale increases, HuBERT\-ECG utilizes 5\-second windows to prioritize training throughput\[[4](https://arxiv.org/html/2608.12695#bib.bib11)\], suggesting that most contemporary SSL strategies assume diagnostically sufficient information is contained within very short temporal horizons\. However, this assumption is challenged by the fact that such narrow windows are often insufficient for definitive clinical rhythm diagnosis, particularly for conditions like AFib\[[17](https://arxiv.org/html/2608.12695#bib.bib3)\]\.

While short temporal windows remain the current standard in ECG modeling, a trend toward signal discretization and tokenization is emerging\. Following the successes of large\-scale language modeling, models such as HeartBERT and HeartLang treat heartbeats as discrete words/tokens\. HeartBERT utilizes Lloyd\-Max quantization and Byte Pair Encoding\[[16](https://arxiv.org/html/2608.12695#bib.bib12)\], while HeartLang employs VQ to construct compact, reusable representations of cardiac morphology\[[7](https://arxiv.org/html/2608.12695#bib.bib13)\]\. While these methods facilitate the use of Transformer\-style sequence modeling, discretization does impose a symbolic bottleneck\.

## IIIMethods

Our overall analytical workflow is illustrated in Fig\.[1](https://arxiv.org/html/2608.12695#S3.F1)\. Preprocessed ECG recordings are segmented into fixed\-length temporal windows and mapped into sequences of latent tokens using either continuous CNN patch embeddings or discrete VQ codebook assignments\. These token sequences are processed by a shared Transformer encoder to model long\-range temporal dependencies and extract contextualized representations\. Global mean pooling aggregates token\-level features into a single window\-level embedding, which is projected into a low\-dimensional latent space via a multilayer projection head\. During SSL pretraining, embeddings from paired windows of the same patient are optimized using an InfoNCE contrastive objective to encourage patient\-consistent representations\.

![Refer to caption](https://arxiv.org/html/2608.12695v1/archi.png)Fig\. 1:Overview of the SSL framework\.Raw ECG signals are mapped to a latent space via CNN patches or VQ tokens\. These sequences are processed by a Transformer to produce patient\-consistent embeddings via an InfoNCE contrastive loss\.Dataset and Preprocessing: All experiments were conducted on the Icentia11k ambulatory*single\-lead*ECG dataset sampled atfs=250f\_\{s\}=250Hz, comprising recordings from 11,000 patients\[[17](https://arxiv.org/html/2608.12695#bib.bib3)\]\. For each patient, the dataset contains on the order of 50 segments of roughly 70 minutes each, collected across daily life\. Segments from the same patient can be separated by days and may reflect different activity states and device placements, making patient\-consistent representation learning substantially more challenging than in short, controlled clinical recordings\. We focus on a single\-lead setting, as recent work reports significant performance degradation when moving from multi\-lead to single\-lead ECGs, highlighting the need for robust representation learning under this constraint\[[10](https://arxiv.org/html/2608.12695#bib.bib9)\]\.

Each record was bandpass filtered using a zero\-phase 4th\-order Butterworth filter with cutoff frequencies of 0\.5–40 Hz\. ECG segments were normalized using per\-window z\-score normalization prior to model input\. All data splits were performed strictly at the patient level\. Patients were randomly shuffled and partitioned into SSL pretraining \(80%\), supervised fine\-tuning \(10%\), validation \(5%\), and test \(5%\) sets, yielding 8,800 patients used exclusively for SSL pretraining\.

Window Sampling and Labeling: All windows were sampled at the patient level to reduce correlation between examples and to control the number of samples contributed per subject\.

Rhythm annotations were converted into non\-overlapping labeled time intervals for normal rhythm \(N\), AFib, and AFL\. Any signal time not covered by known rhythm labels was treated as unlabeled\. Fixed\-length windows were labeled using a strict coverage rule to minimize label noise: a window was considered labelable only if known rhythm labels covered at least 95% of its duration\. Among labelable windows, a window was assigned a positive AFib/AFL label if combined AFib\+AFL occupancy exceeded 5% of the window duration; otherwise, it was labeled as N\. Windows failing the coverage criterion were excluded from supervised analyses\.

Temporal Context and Tokenization: We evaluate four temporal contexts: 16\-second windows \(T=4,000T=4\{,\}000samples\), 1\-minute windows \(T=15,000T=15\{,\}000samples\), 5\-minute windows \(T=75,000T=75\{,\}000samples\), and 10\-minute windows \(T=150,000T=150\{,\}000samples\)\. All use non\-overlapping 160\-sample patches \(0\.64 s\), resulting in 25, 93, 468, and 937 tokens, respectively\.

We compare two ECG tokenization strategies\. In the continuous setting, ECG signals are tokenized using a one\-dimensional CNN patch embedding with kernel size and stride both set to 160 samples, producing a sequence of learned continuous patch embeddings\. In the discretized setting, ECG windows are partitioned into 160\-sample patches and mapped to discrete tokens using VQ\. A K\-means codebook withK=256K=256centroids \(shape256×160256\\times 160\) was trained offline using patches sampled from the SSL patient split\. 64 patches were sampled per patient from a single selected window after per\-window normalization, yielding approximately 563,000 patches\. During training, the codebook was fixed and patch assignments were mapped via a learnable embedding table\.

Model Architecture and SSL Pretraining: The proposed framework, in Fig\.[1](https://arxiv.org/html/2608.12695#S3.F1), utilizes a shared Transformer encoder to process both continuous and discretized input sequences\. The encoder comprises 6 layers with 8 attention heads, hidden dimensiond=256d=256, feedforward expansion factor 4, Gaussian Error Linear Unit \(GELU\) activations, and a dropout rate of 0\.1\. Learned positional embeddings were added to token sequences, and global representations were obtained via mean pooling\. A two\-layer projection head \(256→256→128256\\rightarrow 256\\rightarrow 128\) produced the final L2\-normalized embedding\.

Self\-supervised pretraining used an InfoNCE contrastive objective\[[14](https://arxiv.org/html/2608.12695#bib.bib15)\]with a two\-view batching strategy\. Each training step samplesB=32B=32patients uniformly at random\. For each patient, we draw two distinct windows from different recorded segments to form positive pairs, yielding a combined batch of2​B=642B=64windows processed simultaneously by the encoder\. For each window \(anchor\), the other view from the same patient is treated as the positive, and the remaining2​B−2=622B\-2=62windows \(from other patients\) serve as negatives\. Similarities are computed using cosine similarity with temperatureτ=0\.1\\tau=0\.1, and self\-similarities are masked\. Models were optimized using AdamW with learning rate3×10−43\\times 10^\{\-4\}and weight decay 0\.05, using a cosine learning rate schedule with 200 warmup steps and minimum learning rate10−510^\{\-5\}\. Gradients were clipped to a maximum norm of 1\.0\.

Downstream Evaluation Protocol: Clinical utility was evaluated via downstream AFib/AFL vs\. N classification\. To mitigate class imbalance during supervised fine\-tuning, training windows were sampled with balanced class probability \(p⁡\(AFib/AFL\)=0\.5p\(\\text\{AFib/AFL\}\)=0\.5\)\. Performance on validation and test sets was evaluated using the natural rhythm prevalence, corresponding to approximately 7% positive windows\.

We evaluate both linear probing and end\-to\-end fine\-tuning\. For linear probing, the pretrained encoder was frozen and a linear classifier was trained using AdamW \(learning rate10−310^\{\-3\}, weight decay10−410^\{\-4\}\) with early stopping based on validation Area Under the Precision\-Recall curve \(AUPRC\)\. The same procedure was followed for fine\-tuning, but the encoder and classifier parameters were optimized jointly using AdamW with separate learning rates \(10−410^\{\-4\}for the encoder,10−310^\{\-3\}for the classifier\)\. All supervised models were trained for 20 epochs with a batch size of 64\. For the intermediate 1\-minute and 5\-minute settings, we focus on frozen\-encoder evaluation to measure how SSL representation quality changes with temporal context, rather than repeating the fine\-tuning ablations\.

Performance was measured using AUPRC and Area Under the Receiver Operating Characteristic curve \(AUC\)\. Given the natural class imbalance of the rhythm labels, AUPRC is prioritized as a more sensitive indicator of classification performance for the rare AFib/AFL classes\[[15](https://arxiv.org/html/2608.12695#bib.bib16)\]\. Patient\-level representation quality was assessed using retrieval\-based Recall@1 and Recall@5 on the held\-out test set of 550 patients\. For each patient, we sampled five windows from five randomly selected recording segments, yielding 2,750 test windows in total\. Each window was embedded independently\. For each query window, we retrieved its top\-kknearest neighbors under cosine similarity betweenℓ2\\ell\_\{2\}\-normalized embeddings \(excluding the query itself\)\. A retrieval was considered correct if at least one of thekknearest neighbors corresponded to a window from the same patient\. We use Recall@k for patient retrieval because it is a standard rank\-based identification metric in ECG biometrics and directly measures whether embeddings preserve patient identity among nearest neighbors\[[3](https://arxiv.org/html/2608.12695#bib.bib14)\]\.

All experiments used the same train/validation/test splits and the same evaluation windows by fixing the random seed for splitting and deterministic window sampling\. For supervised training, we sample one labeled window per patient \(1100 patients→\\rightarrow1100 training windows\)\. For validation and testing, we sample two labeled windows per patient and keep this sampling fixed across all models\. All models were trained with identical hyperparameters and stopping criteria\. The only difference across context settings is the duration of the input ECG window\. Because longer windows contain more samples per example, self\-supervised models were trained*until convergence*\(based on plateauing Recall@1/Recall@5\) to avoid favoring any context length due to a fixed step budget\.

TABLE I:Comparative performance of 16\-second and 10\-minute encoders\. \(Agg\.\) denotes 592s window aggregation for the 16\-second encoder; FT and Frozen denote fine\-tuning and linear probing, respectively\.To compare shorter\-context and 10\-minute encoders under a unified clinical evaluation, we evaluate all models on fixed 10\-minute windows\. Encoders trained on shorter contexts are applied independently to non\-overlapping sub\-windows, and their predicted probabilities are averaged\. For the 16\-second encoder, because 600 s is not an integer multiple of 16 s, we discard the final 8 s, yielding 592 s \(148,000 samples\) per window, which decomposes cleanly into 37 input sub\-windows\. The 1\-minute and 5\-minute encoders are evaluated on 10 and 2 non\-overlapping sub\-windows, respectively\. Each sub\-window produces logits for AFib/AFL vs\. N, which are converted to probabilities via softmax and then aggregated at the probability level:p\+=1K​∑j=1Ksoftmax​\(ℓj\)\+p\_\{\+\}=\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\mathrm\{softmax\}\(\\ell\_\{j\}\)\_\{\+\}, whereKKis the number of sub\-windows\. These aggregated probabilities are used directly as decision scores for AUC and AUPRC computation\. We use mean aggregation because AFib/AFL reflect sustained rhythm patterns rather than sparse transient markers; prior work on ECG foundation models similarly finds mean aggregation well\-suited for rhythm\-like labels and interpretable as an ensemble over segments\[[11](https://arxiv.org/html/2608.12695#bib.bib8)\]\.

Experimental Setup: All software, including data preprocessing, ML training, and evaluation were coded in Python language using standard Python libraries, includingNumpy,SciPy,Pandas, andPyTorch\. All experiments were performed using an Ubuntu workstation including two Nvidia RTX 4090 graphics processing units\.

## IVResults & Discussion

\(a\)Temporal scaling of frozen SSL encoders with CNN and VQ tokenization\.\(b\)Discretized VQ encoder embeddings\.\(c\)Continuous CNN encoder embeddings\.
Fig\. 2:The effects of temporal context length and discrete/continuous tokenization\.\(a\) Frozen SSL encoder performance with varying temporal context lengths \(i\.e\., 16\-second, 1\-minute, 5\-minute, and 10\-minute contexts\)\. Evaluation metrics were computed using fixed 10\-minute windows, with shorter\-context models aggregated over non\-overlapping sub\-windows\. \(b\) t\-SNE embeddings of the SSL model trained with the VQ tokenizer, and \(c\) t\-SNE embeddings of the SSL model trained with the CNN tokenizer, trained with a 10\-minute context length, shown for 40 patients colored by patient ID\.Classification Performance: Table[I](https://arxiv.org/html/2608.12695#S3.T1)compares downstream AFib/AFL vs\. N classification results for 16\-second and 10\-minute models\. As explained previously, we additionally perform long\-window aggregation of 37 windows \(16×37=59216\\times 37=592s\) for the 16\-second encoders to enable direct comparison against 10\-minute models\. Across all settings, SSL pretraining substantially improves clinical performance\. SSL models with frozen encoders \(linear probing\) consistently outperform randomly initialized models trained end\-to\-end, demonstrating that SSL learns clinically meaningful representations that transfer effectively with minimal supervision\. For example, in the 10\-minute setting,SSL\+CNN \(Frozen\)achieves an AUPRC of 0\.877 compared to 0\.801 forRandom\+CNN \(FT\)\. Fine\-tuning the encoder further improves peak classification performance for SSL models\. The best overall results are achieved bySSL\+CNN \(FT\), with AUPRC of 0\.946 for 16\-second evaluation and 0\.960 for 10\-minute evaluation\. These results confirm that SSL provides a strong initialization that benefits both linear probing and task\-specific adaptation\.

Impact of Temporal Context: Comparing the shortest and longest settings, 10\-minute pretraining yields consistent improvements over 16\-second pretraining across both clinical and representation\-level evaluations\. Notably, models pretrained on 10\-minute windows outperform 16\-second models when each is evaluated on its native input duration, suggesting meaningful benefits from extended temporal context\. In addition, when 16\-second models are evaluated on 10\-minute windows via aggregation, they recover less of the long\-range structure than models trained directly on 10\-minute context\. Fig\.[2\(a\)](https://arxiv.org/html/2608.12695#S4.F2.sf1)further extends this comparison by adding 1\-minute and 5\-minute contexts under the same fixed 10\-minute aggregation protocol, showing that the largest gains emerge once the input window reaches 5 and 10 minutes\. These findings support the hypothesis that extended temporal context enables integration of rhythm dynamics, baseline morphology, and patient\-specific characteristics that are not fully captured in short snapshots\.

Continuous vs\. Discretized Tokenization: Continuous CNN encoders demonstrate clear superiority over discretized VQ models across all evaluated settings, in both clinical utility and patient\-level retrieval\. For example, in the 10\-minute frozen encoder setting,SSL\+CNNachieves an AUPRC of 0\.877 and Recall@1 of 0\.907, compared to 0\.610 and 0\.351 forSSL\+VQ, respectively\. Similar trends are observed across fine\-tuned, 16\-second, and intermediate\-context settings\. These results indicate that discretization imposes a quantization bottleneck that degrades both fine\-grained morphological modeling and preservation of patient\-specific structure\. Continuous encoders more effectively capture subtle waveform characteristics critical for both clinical task performance and patient\-level consistency in representation learning\.

Patient\-Level Retrieval and Representation Quality: Longer\-context SSL models achieve markedly higher retrieval performance, withSSL\+CNN \(Frozen, 10\-minute\)reaching Recall@1 of 0\.907 and Recall@5 of 0\.945\. In contrast, 16\-second SSL models achieve lower but still substantial retrieval performance \(e\.g\., Recall@1 of 0\.737 forSSL\+CNN \(Frozen, 16\-second\)\), while randomly initialized models perform near chance\. This confirms that high retrieval accuracy is not a trivial artifact of the evaluation protocol, but reflects meaningful patient\-consistent structure learned through SSL\. The intermediate\-context results further show that retrieval improves as temporal context increases\. These results demonstrate that extended\-context self\-supervised learning is critical for learning representations that remain consistent across recording sessions, activities, and temporal separation\.

Effect of Fine\-Tuning on Representation Structure: While fine\-tuning improves clinical classification performance, it slightly reduces patient\-level retrieval accuracy for SSL models\. For example, in the 10\-minuteSSL\+CNNsetting, Recall@1 decreases from 0\.907 \(frozen\) to 0\.851 \(fine\-tuned\)\. This indicates a trade\-off between task\-specific specialization and preservation of general patient\-consistent structure\. Fine\-tuning biases representations toward features most predictive of AFib/AFL, distorting pretrained features and partially reducing identity\-preserving information\. This behavior is consistent with prior theoretical and empirical findings showing that supervised fine\-tuning can distort and compress pretrained representations in favor of task\-relevant features\[[9](https://arxiv.org/html/2608.12695#bib.bib17)\]\.

Visualization of Long\-Context Representations: Fig\.[2\(b\)](https://arxiv.org/html/2608.12695#S4.F2.sf2)and[2\(c\)](https://arxiv.org/html/2608.12695#S4.F2.sf3)visualize 10\-minute frozen SSL embeddings using t\-SNE\[[18](https://arxiv.org/html/2608.12695#bib.bib18)\]for discretized VQ and continuous CNN tokenization, respectively\. The SSL\+CNN embeddings exhibit substantially tighter and more coherent patient\-specific clusters compared to SSL\+VQ, which shows increased overlap and fragmentation\. This visualization is qualitative, but it is consistent with the quantitative Recall@k results and illustrates that continuous encoders trained on long temporal context better preserve patient\-consistent structure than discretized representations\.

## VConclusion

We investigated how temporal context and tokenization choices affect self\-supervised representation learning on ambulatory ECG recordings\. Across linear probing and end\-to\-end fine\-tuning, extended\-context pretraining improves AFib/AFL classification and substantially strengthens patient\-level retrieval compared to 16\-second snapshots, with the largest gains observed for 5\- and 10\-minute contexts\. This indicates better preservation of longitudinal, patient\-level structure\. We further find that continuous CNN tokenization outperforms discretized VQ across clinical metrics and retrieval, suggesting a discretization bottleneck that leads to morphological blurring\. Collectively, these results provide practical guidance for ECG foundation model design: incorporating extended temporal context and continuous encoders yields representations that transfer better to clinical tasks and remain stable across recording sessions, supporting downstream uses, such as similarity search, cohort stratification, and longitudinal monitoring\. Ethics Statement:Our experiments utilized publicly available datasets and no data was collected as part of the study\. As such, institutional approval was not required\.

Acknowledgments:This study was partially supported by the U\.S\. National Science Foundation grant IIS\-2337909\.

## References

- \[1\]Z\. I\. Attia, S\. Kapa, F\. Lopez\-Jimenez, P\. M\. McKie, D\. J\. Ladewig, G\. Satam, P\. A\. Pellikka, M\. Enriquez\-Sarano, P\. A\. Noseworthy, T\. M\. Munger,et al\.\(2019\)Screening for cardiac contractile dysfunction using an artificial intelligence–enabled electrocardiogram\.Nature medicine25\(1\),pp\. 70–74\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p1.1)\.
- \[2\]Z\. I\. Attia, P\. A\. Noseworthy, F\. Lopez\-Jimenez, S\. J\. Asirvatham, A\. J\. Deshmukh, B\. J\. Gersh, R\. E\. Carter, X\. Yao, A\. A\. Rabinstein, B\. J\. Erickson,et al\.\(2019\)An artificial intelligence\-enabled ecg algorithm for the identification of patients with atrial fibrillation during sinus rhythm: a retrospective analysis of outcome prediction\.The Lancet394\(10201\),pp\. 861–867\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p1.1)\.
- \[3\]\(2017\)Comparative analysis of bag\-of\-words models for ecg\-based biometrics\.IET Biometrics6\(6\),pp\. 495–502\.Cited by:[§III](https://arxiv.org/html/2608.12695#S3.p12.1)\.
- \[4\]E\. Coppola, M\. Savardi, M\. Massussi, M\. Adamo, M\. Metra, and A\. Signoroni\(2024\)HuBERT\-ecg as a self\-supervised foundation model for broad and scalable cardiac applications\.medRxiv,pp\. 2024–11\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p2.1),[§II](https://arxiv.org/html/2608.12695#S2.p2.1)\.
- \[5\]N\. Diamant, E\. Reinertsen, S\. Song, A\. D\. Aguirre, C\. M\. Stultz, and P\. Batra\(2022\)Patient contrastive learning: a performant, expressive, and practical approach to electrocardiogram modeling\.PLoS computational biology18\(2\),pp\. e1009862\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p1.1)\.
- \[6\]A\. Y\. Hannun, P\. Rajpurkar, M\. Haghpanahi, G\. H\. Tison, C\. Bourn, M\. P\. Turakhia, and A\. Y\. Ng\(2019\)Cardiologist\-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network\.Nature medicine25\(1\),pp\. 65–69\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p1.1)\.
- \[7\]J\. Jin, H\. Wang, H\. Li, J\. Li, J\. Pan, and S\. Hong\(2025\)Reading your heart: learning ECG words and sentences via pre\-training ECG language model\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6Hz1Ko087B)Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p2.1),[§II](https://arxiv.org/html/2608.12695#S2.p3.1)\.
- \[8\]D\. Kiyasseh, T\. Zhu, and D\. A\. Clifton\(2021\)Clocs: contrastive learning of cardiac signals across space, time, and patients\.InInternational Conference on Machine Learning,pp\. 5606–5615\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p4.1),[§II](https://arxiv.org/html/2608.12695#S2.p1.1)\.
- \[9\]A\. Kumar, A\. Raghunathan, R\. M\. Jones, T\. Ma, and P\. Liang\(2022\)Fine\-tuning can distort pretrained features and underperform out\-of\-distribution\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=UYneFzXSJWh)Cited by:[§IV](https://arxiv.org/html/2608.12695#S4.p5.1)\.
- \[10\]J\. Li, A\. Aguirre, J\. Moura, C\. Liu, L\. Zhong, C\. Sun, G\. Clifford, M\. B\. Westover, and S\. Hong\(2024\)An electrocardiogram foundation model built on over 10 million recordings with external evaluation across multiple domains\.External Links:2410\.04133,[Link](https://arxiv.org/abs/2410.04133)Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p1.1),[§III](https://arxiv.org/html/2608.12695#S3.p2.1)\.
- \[11\]K\. McKeen, S\. Masood, A\. Toma, B\. Rubin, and B\. Wang\(2025\)ECG\-fm: an open electrocardiogram foundation model\.External Links:2408\.05178,[Link](https://arxiv.org/abs/2408.05178)Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p1.1),[§III](https://arxiv.org/html/2608.12695#S3.p14.1)\.
- \[12\]T\. Mehari and N\. Strodthoff\(2022\)Self\-supervised representation learning from 12\-lead ecg data\.Computers in biology and medicine141,pp\. 105114\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p1.1),[§I](https://arxiv.org/html/2608.12695#S1.p2.1),[§II](https://arxiv.org/html/2608.12695#S2.p1.1)\.
- \[13\]Y\. Na, M\. Park, Y\. Tae, and S\. Joo\(2024\)Guiding masked representation learning to capture spatio\-temporal relationship of electrocardiogram\.arXiv preprint arXiv:2402\.09450\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p2.1),[§II](https://arxiv.org/html/2608.12695#S2.p2.1)\.
- \[14\]A\. v\. d\. Oord, Y\. Li, and O\. Vinyals\(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§III](https://arxiv.org/html/2608.12695#S3.p9.1)\.
- \[15\]T\. Saito and M\. Rehmsmeier\(2015\)The precision\-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets\.PloS one10\(3\),pp\. e0118432\.Cited by:[§III](https://arxiv.org/html/2608.12695#S3.p12.1)\.
- \[16\]S\. Tahery, F\. H\. Akhlaghi, and T\. Amirsoleimani\(2024\)HeartBERT: a self\-supervised ecg embedding model for efficient and effective medical signal analysis\.arXiv preprint arXiv:2411\.11896\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p2.1),[§II](https://arxiv.org/html/2608.12695#S2.p3.1)\.
- \[17\]S\. Tan, G\. Androz, A\. Chamseddine, P\. Fecteau, A\. Courville, Y\. Bengio, and J\. P\. Cohen\(2019\)Icentia11k: an unsupervised representation learning dataset for arrhythmia subtype discovery\.arXiv preprint arXiv:1910\.09570\.Cited by:[§I](https://arxiv.org/html/2608.12695#S1.p4.1),[§II](https://arxiv.org/html/2608.12695#S2.p1.1),[§II](https://arxiv.org/html/2608.12695#S2.p2.1),[§III](https://arxiv.org/html/2608.12695#S3.p2.1)\.
- \[18\]L\. van der Maaten and G\. Hinton\(2008\)Visualizing data using t\-sne\.Journal of Machine Learning Research9\(86\),pp\. 2579–2605\.External Links:[Link](http://jmlr.org/papers/v9/vandermaaten08a.html)Cited by:[§IV](https://arxiv.org/html/2608.12695#S4.p6.1)\.