Beyond Gene Reconstruction: Learning Cell Representations through Complementary Transcriptomic Views

# Beyond Gene Reconstruction: Learning Cell Representations through Complementary Transcriptomic Views
Source: [https://arxiv.org/html/2608.00985](https://arxiv.org/html/2608.00985)
Jiaqi Xiong\\equalcontrib2, Yuntao Hu\\equalcontrib3, Yu Zheng4, Yifei Shi5, Xinyue Guo2, Jiaxin Qi\\corresponding1

###### Abstract

The rapid growth of single\-cell transcriptomic data has enabled the development of foundation models pretrained primarily by reconstructing masked expression values\. This objective encourages these models to learn gene dependencies but does not directly optimize whole\-cell representations, which are essential for many downstream tasks\. To bridge this gap, we propose a contrastive pretraining framework that learns cell representations through complementary transcriptomic views\. Since standard contrastive learning is not readily applicable to single\-cell pretraining, we introduce specific adaptations along three dimensions — co\-expression\-guided gene partitioning, expression\-aware contrast\-set construction, and competence\-gated contrastive onset\. Specifically, we first construct two complementary views of each cell by partitioning its genes according to their co\-expression structure\. Then, to prevent the model from using gene\-set identity as a shortcut, we construct hard negatives by permuting expression values while keeping gene identities unchanged\. Finally, we introduce a competence\-aware controller to determine how the contrastive objective is applied\. Experiments on cell\-type annotation and gene regulatory network inference demonstrate competitive transfer under the evaluated protocols\. In the six\-network GRN evaluation, our method records the highest mean AUROC and AUPRC point estimates among the compared variants, while the highest\-scoring variant differs across individual networks\. These results establish complementary\-view contrastive learning as an effective direction for single\-cell pretraining beyond gene reconstruction\.

## Introduction

![Refer to caption](https://arxiv.org/html/2608.00985v1/x1.png)Figure 1:Motivation forCoCoS\. \(a\) Masked expression\-value prediction directly supervises reconstruction through gene\-level features but does not impose a metric objective on the whole\-cell representation\. \(b\) Naively transferring contrastive learning raises two transcriptome\-specific challenges: unconstrained expression perturbations may alter the cellular state used to define a positive pair, while negatives with different retained genes may be separated by gene identity rather than gene–value correspondence\.Large\-scale single\-cell transcriptomic data have motivated the development of foundation models that are predominantly pretrained through masked expression\-value prediction\(Cuiet al\.[2024](https://arxiv.org/html/2608.00985#bib.bib3); Theodoriset al\.[2023](https://arxiv.org/html/2608.00985#bib.bib4); Yanget al\.[2022a](https://arxiv.org/html/2608.00985#bib.bib5); Haoet al\.[2024](https://arxiv.org/html/2608.00985#bib.bib6)\)\. By reconstructing masked values, these models learn gene\-level dependencies and capture latent co\-expression structure\. However, this objective provides direct supervision only at individual genes, and a masked expression value can often be inferred from a limited neighborhood of correlated genes rather than from a representation of the complete cellular state\(Xionget al\.[2025](https://arxiv.org/html/2608.00985#bib.bib2); Kotliaret al\.[2019](https://arxiv.org/html/2608.00985#bib.bib34)\)\. As a result, strong reconstruction performance does not necessarily translate into discriminative and transferable whole\-cell representations, even though such representations are essential for downstream applications including cell\-type annotation and gene regulatory network inference\.

This objective mismatch makes cell\-level contrastive supervision a natural complement to masked reconstruction\. However, conventional contrastive learning does not transfer straightforwardly to single\-cell transcriptomes, where two domain\-specific challenges arise, as illustrated in Figure[1](https://arxiv.org/html/2608.00985#Sx1.F1)\. First, constructing biologically valid positive pairs is nontrivial\. Unlike image augmentations such as cropping, resizing, and color jitter, which can alter low\-level appearance while preserving object identity\(Chenet al\.[2020](https://arxiv.org/html/2608.00985#bib.bib7)\), perturbing gene\-expression values may change cell identity or dynamic biological programs rather than preserve the same biological state\(Kotliaret al\.[2019](https://arxiv.org/html/2608.00985#bib.bib34)\)\. Second, negative\-set construction is vulnerable to gene\-identity shortcuts\. Conventional contrastive learning typically treats other samples in the minibatch as negatives\. In sparse transcriptomic inputs, however, expression sparsity and sequence truncation may cause different cells to retain different subsets of gene identities\. A model can therefore distinguish negative cells from the anchor using gene\-set composition alone, without learning the correspondence between genes and their expression values\.

To address these challenges, we introduceCoCoS, a contrastive pretraining framework tailored to single\-cell transcriptomes\. To construct biologically meaningful positive pairs without altering cell identity,CoCoSkeeps measured gene–value pairs unchanged and uses a co\-expression\-guided gene partition to define two complementary view templates\. The resulting inputs remain anchored to the same measured cell, avoiding artificial expression shifts while exposing complementary biological evidence\. To prevent the gene\-identity shortcut during contrast\-set construction,CoCoScombines expression\-aware sampling with same\-view negatives and identity\-matched hard negatives generated by fixing gene identities and permuting their associated expression values\. Because the original and permuted inputs contain the same genes, gene\-set composition alone is insufficient, encouraging discrimination based on gene–value correspondence\. We refer to these components as*Co\-expression\-Guided Gene Partitioning*and*Expression\-Aware Contrast\-Set Construction*, respectively; their motivating challenges are summarized in Figure[1](https://arxiv.org/html/2608.00985#Sx1.F1)\.

While these two components determine what the model contrasts, effective pretraining also depends on when contrastive supervision is introduced\. Because the complementary panels share no gene tokens, relating them requires the encoder to recognize that different sets of genes reflect the same underlying biological programs\. Early in pretraining, however, these cross\-gene dependencies have not yet been learned through masked expression reconstruction\. Applying the contrastive objective immediately may therefore encourage the model to align the panels using simpler technical cues, such as library size or expression sparsity, rather than their biological correspondence\. To address this problem,CoCoSintroduces*Competence\-Gated Contrastive Onset*\. The model first learns gene\-level dependencies through masked expression modeling\. Once label\-free diagnostics on held\-out cells indicate stable reconstruction competence and cross\-view structure, the contrastive objective is introduced gradually\. Together, the three components address the key challenges of positive\-view construction, negative\-set design, and contrastive onset, enabling cell\-level contrastive pretraining from complementary transcriptomic views\.

Experiments across downstream single\-cell tasks demonstrate competitive transfer under the evaluated protocols\. Two controlled analyses characterize view sampling and contrastive scheduling, supporting the effectiveness ofCoCoSfor whole\-cell representation learning in single\-cell transcriptomic pretraining\.

Our main contributions are:

- •We identify the mismatch between masked expression\-value reconstruction and whole\-cell representation learning, and characterize three requirements for applying cell\-level contrast to single\-cell pretraining: biologically valid positive views, gene\-identity shortcuts in negative samples, and the timing of contrastive supervision\.
- •We introduceCoCoS, which addresses these requirements through co\-expression\-guided gene partitioning, expression\-aware contrast\-set construction, and competence\-gated contrastive onset\.
- •We conduct extensive experiments across cell\-type annotation and gene regulatory network inference to evaluate the effectiveness ofCoCoS\. Results demonstrate that properly bounded whole\-cell contrastive supervision yields more discriminative and transferable cell representations than pure reconstruction\-based methods\.

## Related Work

#### Single\-cell foundation models\.

Masked reconstruction is common across biological foundation models: DNABERT and the Nucleotide Transformer predict masked DNAkk\-mers, MutBERT models probabilistic genomic variation, and RNA\-FM predicts masked RNA tokens\(Jiet al\.[2021](https://arxiv.org/html/2608.00985#bib.bib36); Dalla\-Torreet al\.[2025](https://arxiv.org/html/2608.00985#bib.bib37); Longet al\.[2025](https://arxiv.org/html/2608.00985#bib.bib38); Chenet al\.[2022](https://arxiv.org/html/2608.00985#bib.bib39)\)\. Their losses act at corrupted sequence positions, although the learned representations transfer to sequence\-level tasks\. Single\-cell profiles instead pair gene identities with expression measurements in sparse cellular states\(Kotliaret al\.[2019](https://arxiv.org/html/2608.00985#bib.bib34)\)\. Large expression corpora have enabled scBERT, Geneformer, scGPT, and scFoundation\(Yanget al\.[2022a](https://arxiv.org/html/2608.00985#bib.bib5); Theodoriset al\.[2023](https://arxiv.org/html/2608.00985#bib.bib4); Cuiet al\.[2024](https://arxiv.org/html/2608.00985#bib.bib3); Haoet al\.[2024](https://arxiv.org/html/2608.00985#bib.bib6)\), with recent benchmark work examining their transfer behavior\(Qiet al\.[2025](https://arxiv.org/html/2608.00985#bib.bib1)\)\. These models use masked gene or expression prediction and generative modeling to learn gene dependencies, but do not explicitly organize whole\-cell embedding geometry\.

#### Single\-cell representation learning\.

In addition to foundation models, dedicated representation\-learning methods optimize cell embeddings for clustering, integration, and atlas mapping\. Contrastive clustering, CLEAR, and Concerto learn from augmented profiles or paired encoder views without incorporating contrast into a masked\-expression foundation\-model objective\(Ciortan and Defrance[2021](https://arxiv.org/html/2608.00985#bib.bib29); Hanet al\.[2022](https://arxiv.org/html/2608.00985#bib.bib30); Yanget al\.[2022b](https://arxiv.org/html/2608.00985#bib.bib31)\)\. More recent large\-scale approaches bring cell\-level contrast into pretraining in different forms\. LangCell combines masked\-gene modeling with intra\- and inter\-modal contrastive objectives; TABULA jointly performs column\-wise gene reconstruction and row\-wise cell contrastive learning; and scConcept replaces reconstruction with contrastive pretraining over disjoint gene panels\(Zhaoet al\.[2024](https://arxiv.org/html/2608.00985#bib.bib21); Dinget al\.[2025](https://arxiv.org/html/2608.00985#bib.bib32); Bahramiet al\.[2025](https://arxiv.org/html/2608.00985#bib.bib22)\)\. Cell\-level contrast is therefore not itself new\.CoCoSinstead retains masked expression\-value prediction and jointly designs co\-expression\-guided views, expression\-aware contrast sets with fixed\-identity value derangements, and a data\-dependent contrastive onset for this setting\.

#### Adaptive training\.

Curricula order or filter examples by difficulty, with competence\-based and adaptive variants changing exposure according to training progress or model state\(Bengioet al\.[2009](https://arxiv.org/html/2608.00985#bib.bib13); Platanioset al\.[2019](https://arxiv.org/html/2608.00985#bib.bib14); Konget al\.[2021](https://arxiv.org/html/2608.00985#bib.bib15)\)\. GradNorm and Auto\-λ\\lambdainstead continuously reweight active losses\(Chenet al\.[2018](https://arxiv.org/html/2608.00985#bib.bib12); Liuet al\.[2022](https://arxiv.org/html/2608.00985#bib.bib16)\)\.CoCoScontrols a different variable: a fixed, held\-out, label\-free cohort triggers a previously inactive contrastive objective while masked\-value reconstruction remains active\. This is data\-dependent objective onset rather than sample curriculum or generic multitask loss balancing\.

![Refer to caption](https://arxiv.org/html/2608.00985v1/x2.png)Figure 2:Overview ofCoCoS\-GPC\. Co\-expression\-guided partitioning and expression\-aware sampling form two disjoint views, which are masked and reconstructed by a shared Transformer\. The paired same\-cell views define the positive, while same\-view and cross\-view in\-batch examples together with fixed\-identity value derangements form the contrast set\. A label\-free sentinel tracks smoothed gene\-program concordance, reconstruction error, and representation repertoire; once readiness persists, the controller activates the contrastive objective and ramps its weight, which remains zero beforehand\.

## Method

Masked\-value reconstruction directly supervises individual gene values but not the geometry of the whole\-cell representation\.CoCoSaugments masked\-value reconstruction with three coordinated operations\. Co\-expression\-guided gene partitioning forms complementary views; expression\-aware contrast\-set construction combines weighted sampling with identity\-matched hard negatives; and competence\-gated contrastive onset activates cell\-level alignment only after the masked predictor satisfies a label\-free readiness criterion\. The resulting instantiation isCoCoS\-GPC\. Figure[2](https://arxiv.org/html/2608.00985#Sx2.F2)summarizes the framework, and the following sections introduce the backbone and these operations in training order\.

### Backbone and Masked Pretraining

Letℳ=\{\(𝐠i,𝐱i\)\}i=1N\\mathcal\{M\}=\\\{\(\\mathbf\{g\}\_\{i\},\\mathbf\{x\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}denote the pretraining dataset ofNNcells\. For cellii,𝐠i=\(gi​1,gi​2,…,gi​ni\)\\mathbf\{g\}\_\{i\}=\(g\_\{i1\},g\_\{i2\},\\ldots,g\_\{in\_\{i\}\}\)is its gene\-identity sequence, wheregi​j∈𝒢g\_\{ij\}\\in\\mathcal\{G\}and𝒢\\mathcal\{G\}denotes the gene vocabulary\. The aligned expression sequence𝐱i=\(xi​1,xi​2,…,xi​ni\)\\mathbf\{x\}\_\{i\}=\(x\_\{i1\},x\_\{i2\},\\ldots,x\_\{in\_\{i\}\}\)contains the corresponding continuous expression values\.

For masked\-value prediction, we sample a set of masked positionsΩi⊆\{1,…,ni\}\\Omega\_\{i\}\\subseteq\\\{1,\\ldots,n\_\{i\}\\\}and replace the expression values at these positions to obtain the masked sequence𝐱~i\\tilde\{\\mathbf\{x\}\}\_\{i\}\. We use the Transformer backbone from scBenchmark\(Qiet al\.[2025](https://arxiv.org/html/2608.00985#bib.bib1)\)to encode the gene identities and masked expression values\. The masked\-value reconstruction loss in a minibatchBBis

ℒmlm=1∑i=1B\|Ωi\|​∑i=1B∑j∈Ωi\(fθ​\(𝐠i,𝐱~i\)j−xi​j\)2,\\mathcal\{L\}\_\{\\mathrm\{mlm\}\}=\\frac\{1\}\{\\sum\_\{i=1\}^\{B\}\|\\Omega\_\{i\}\|\}\\sum\_\{i=1\}^\{B\}\\sum\_\{j\\in\\Omega\_\{i\}\}\\left\(f\_\{\\theta\}\(\\mathbf\{g\}\_\{i\},\\tilde\{\\mathbf\{x\}\}\_\{i\}\)\_\{j\}\-x\_\{ij\}\\right\)^\{2\},\(1\)wherefθf\_\{\\theta\}denotes the Transformer encoder together with its masked\-value prediction layer, andfθ​\(𝐠i,𝐱~i\)jf\_\{\\theta\}\(\\mathbf\{g\}\_\{i\},\\tilde\{\\mathbf\{x\}\}\_\{i\}\)\_\{j\}is the predicted expression value at positionjj\. However, because this reconstruction objective provides no explicit supervision for the whole\-cell representation, it does not directly optimize the cell embedding for discriminative and transferable downstream use\.

### Our Method:CoCoS

Co\-expression\-Guided Gene Partitioning\. Constructing biologically valid positive pairs is nontrivial in single\-cell pretraining, because perturbing expression values may alter the underlying cellular state\.CoCoSinstead partitions each cell into two complementary gene views and applies the same masking scheme to both, providing a positive pair while preserving the reconstruction objective\.

We construct an undirected weighted co\-occurrence graph\(𝒢,E,w\)\(\\mathcal\{G\},E,w\), wherewg​hw\_\{gh\}counts how often genesggandhhco\-occur among the most highly expressed genes of a training cell\. After retaining each gene’s strongest neighbors, we seek a bipartition that places strongly co\-occurring genes in different views:

maxχ:𝒢→\{0,1\}​∑\{g,h\}∈Ewg​h​1​\{χ​\(g\)≠χ​\(h\)\}\.\\max\_\{\\chi:\\mathcal\{G\}\\to\\\{0,1\\\}\}\\sum\_\{\\\{g,h\\\}\\in E\}w\_\{gh\}\\,\\mathbf\{1\}\\\{\\chi\(g\)\\neq\\chi\(h\)\\\}\.\(2\)We construct the partition with weighted\-degree ordering and count\-based placement\. Genes are visited in decreasing order of their summed incident co\-occurrence weights; each gene is then assigned opposite the panel containing more of its already assigned neighbors, with every neighbor contributing one count irrespective ofwg​hw\_\{gh\}\. For each minibatch, we further reverse each gene’s assignment with a small probability and share the resulting partition across all cells\. For cellii, the partition defines two complementary sets of gene positions,Ωi\\Omega\_\{i\}andΩ¯i\\bar\{\\Omega\}\_\{i\}\. The first view masks the expression values at positions inΩi\\Omega\_\{i\}, whereas the second masks those at positions inΩ¯i\\bar\{\\Omega\}\_\{i\}\. Therefore, each gene value is observed in one view and masked in the other, allowing the two views to provide complementary information while preserving the original gene identities and expression values\. The masked\-value reconstruction loss defined above can be applied to both views\.

Expression\-Aware Contrast\-Set Construction\.Although complementary masking provides positive pairs, standard in\-batch contrast may exploit gene identities as a shortcut\. Different cells in the same minibatch generally contain different subsets of genes, whereas the two views of the same cell preserve the same gene identities\. The model may therefore distinguish positive and negative pairs from gene composition alone, without learning the correspondence between genes and their expression values\.

To prevent this shortcut, we combine expression\-aware sampling with identity\-matched hard negatives\. Specifically, letΩi\{\\Omega\}\_\{i\}denote the positions kept unmasked in one view, and let𝐱i=\(xi​j\)j∈Ωi\{\\mathbf\{x\}\}\_\{i\}=\(x\_\{ij\}\)\_\{j\\in\{\\Omega\}\_\{i\}\}denote the corresponding visible expression values\. We construct an identity\-matched hard negative by permuting the values in𝐱i\{\\mathbf\{x\}\}\_\{i\}while keeping the gene identities𝐠i\\mathbf\{g\}\_\{i\}and the masking pattern unchanged\. The original and permuted views therefore contain identical gene identities but different gene–value assignments, requiring the model to discriminate between them based on gene–value correspondence\. The same construction is applied to the complementary view\.

Let𝐞i\\mathbf\{e\}\_\{i\}denote the embedding of an anchor view,𝐞¯i\\bar\{\\mathbf\{e\}\}\_\{i\}the embedding of its complementary view, andℰi−\\mathcal\{E\}\_\{i\}^\{\-\}the corresponding negative set\. The negative set contains embeddings of other cells in the minibatch from both views, together with the identity\-matched hard\-negative embeddings constructed above\. For either view chosen as the anchor, the contrastive loss is

ℒctr=−1B​∑i=1Blog⁡exp⁡\(𝐞i⊤​𝐞¯i\)exp⁡\(𝐞i⊤​𝐞¯i\)\+∑𝐞−∈ℰi−exp⁡\(𝐞i⊤​𝐞−\)\.\\mathcal\{L\}\_\{\\mathrm\{ctr\}\}=\-\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\log\\frac\{\\exp\\\!\\left\(\\mathbf\{e\}\_\{i\}^\{\\top\}\\bar\{\\mathbf\{e\}\}\_\{i\}\\right\)\}\{\\exp\\\!\\left\(\\mathbf\{e\}\_\{i\}^\{\\top\}\\bar\{\\mathbf\{e\}\}\_\{i\}\\right\)\+\\sum\_\{\\mathbf\{e\}^\{\-\}\\in\\mathcal\{E\}\_\{i\}^\{\-\}\}\\exp\\\!\\left\(\\mathbf\{e\}\_\{i\}^\{\\top\}\\mathbf\{e\}^\{\-\}\\right\)\}\.\(3\)The final objective averages this loss over the two complementary choices of anchor\.

Table 1:Cell\-type annotation under the unified ten\-dataset cell\-forward evaluation protocol:kk\-NN accuracy \(%\) from frozen whole\-cell embeddings, one training seed and one downstream split\. Unshaded rows are external references, light\-gray rows are internal controls, and the darker row is proposed\. Within each column, all entries attaining the highest value are bold and the next distinct value is underlined\.†Geneformer is evaluated on the same cell rows and seed\-42 split after a model\-specific top\-512 median/rank adaptation\. Because this is neither native\-input Geneformer nor numerically interchangeable with the legacy rows, it is shown for coverage but excluded from the bold/underline ranking\.Competence\-Gated Contrastive Onset\. Introducing cell\-level contrast before masked reconstruction has learned stable gene dependencies may encourage the model to align views through technical cues, such as library size or sparsity\. We therefore keep the contrastive weightβ​\(t\)=0\\beta\(t\)=0until a fixed, unlabeled sentinel cohort disjoint from training indicates readiness\. The gate uses neither cell labels nor downstream scores and is not tied to a predefined epoch\.

At probepp, letμp\+\\mu\_\{p\}^\{\+\}denote the mean cosine similarity between paired sentinel views, and letμp−\\mu\_\{p\}^\{\-\}andσp−\\sigma\_\{p\}^\{\-\}denote the mean and standard deviation over mismatched cross\-view pairs\. We define gene\-program concordance \(GPC\) and the repertoire score as

cp=μp\+−μp−max⁡\(σp−,εc\),qp=‖std⁡\(𝐙p\)‖2​erankp,c\_\{p\}=\\frac\{\\mu\_\{p\}^\{\+\}\-\\mu\_\{p\}^\{\-\}\}\{\\max\(\\sigma\_\{p\}^\{\-\},\\varepsilon\_\{c\}\)\},\\qquad q\_\{p\}=\\\|\\operatorname\{std\}\(\\mathbf\{Z\}\_\{p\}\)\\\|\_\{2\}\\sqrt\{\\operatorname\{erank\}\_\{p\}\},\(4\)whereεc\>0\\varepsilon\_\{c\}\>0is a numerical stabilizer,𝐙p\\mathbf\{Z\}\_\{p\}stacks the normalized sentinel embeddings, anderankp\\operatorname\{erank\}\_\{p\}is the entropy effective rank of their centered covariance\. GPC measures the separation between paired and mismatched views relative to off\-pair variation, whileqpq\_\{p\}monitors embedding diversity\.

The same probe also records the pooled masked\-value MSE across both views\. Using early sentinel probes for calibration, we smooth the GPC, reconstruction\-error, and repertoire traces\. The controller declares readiness when cross\-view concordance is stable and embedding diversity is preserved for a prescribed number of consecutive probes\. The first optimizer update satisfying these conditions is denoted byt⋆t^\{\\star\}, after which the controller triggers irreversibly\.

After the trigger, letRRdenote the ramp duration and defineu​\(t\)=clip⁡\(\(t−t⋆\)/R,0,1\)u\(t\)=\\operatorname\{clip\}\(\(t\-t^\{\\star\}\)/R,0,1\)\. The contrastive weight is

β​\(t\)=βmax​sin2⁡\(π2​u​\(t\)\)\.\\beta\(t\)=\\beta\_\{\\max\}\\sin^\{2\}\\\!\\left\(\\frac\{\\pi\}\{2\}u\(t\)\\right\)\.\(5\)Thus,β​\(t\)\\beta\(t\)remains zero beforet⋆t^\{\\star\}and increases smoothly toβmax\\beta\_\{\\max\}afterward\. Identity\-matched hard negatives are constructed only whenβ​\(t\)\>0\\beta\(t\)\>0\.

Overall Objective\. The complete training objective combines masked\-value reconstruction on both complementary views with the gated contrastive objective:

ℒoverall=ℒmlm\+β​\(t\)​ℒctr\.\\mathcal\{L\}\_\{\\mathrm\{overall\}\}=\\mathcal\{L\}\_\{\\mathrm\{mlm\}\}\+\\beta\(t\)\\mathcal\{L\}\_\{\\mathrm\{ctr\}\}\.\(6\)Before the readiness criterion is met, the model is optimized solely through masked\-value reconstruction\. After the trigger, contrastive supervision gradually shapes the whole\-cell embedding while reconstruction remains active on both views\. This design preserves gene\-level dependency learning throughout pretraining and introduces explicit cell\-level supervision\.

## Experiments

### Datasets

#### Pretraining corpus\.

The pretraining corpus contains1,813,7801\{,\}813\{,\}780source rows and follows the data construction of Qi et al\.\(Qiet al\.[2025](https://arxiv.org/html/2608.00985#bib.bib1)\)from the publicly available CELLxGENE collection\. Before training, we reserve three mutually disjoint held\-out cohorts:2,0482\{,\}048rows for readiness assessment,2,0482\{,\}048rows for an independent non\-controlling audit, and10,00010\{,\}000rows for the partition audit\. These cohorts are also disjoint from the training pool, leaving1,799,6841\{,\}799\{,\}684rows for optimization\. The partition\-audit cohort evaluates the fixed gene partition and never affects the contrastive\-onset decision\.

#### Downstream evaluation\.

We evaluate frozen whole\-cell embeddings on the ten cell\-type annotation datasets assembled byQiet al\.\([2025](https://arxiv.org/html/2608.00985#bib.bib1)\)\(Table[1](https://arxiv.org/html/2608.00985#Sx3.T1)\)\. Geneformer V2\-104M\(Theodoriset al\.[2023](https://arxiv.org/html/2608.00985#bib.bib4)\)is evaluated on the same ten cell sets, labels, cell rows, and fixed seed\-42 splits; only its model\-specific input encoding differs, as detailed in Section[Experimental Setup](https://arxiv.org/html/2608.00985#Sx4.SSx2)\. Separately, we score gene\-regulatory edges on six BEELINE Specific 1000\-gene networks \(hESC, hHep, mDC, mHSC\-E, mHSC\-GM, and mHSC\-L\)\(Pratapaet al\.[2020](https://arxiv.org/html/2608.00985#bib.bib41)\), treating all unlabeled candidate pairs as negatives\. Annotation and GRN labels are used only for downstream evaluation; neither enters pretraining or the contrastive\-onset decision\.

Table 2:Linear\-probe test accuracy \(%\) on the same ten cell\-type annotation datasets as Table[1](https://arxiv.org/html/2608.00985#Sx3.T1)\. Evaluation settings and visual conventions follow Table[1](https://arxiv.org/html/2608.00985#Sx3.T1); results use one training seed and one fixed downstream split\.†Geneformer uses a model\-specific top\-512 median/rank adaptation and a frozen classifier evaluated only at epoch 50, whereas the archived legacy rows use their original evaluator\. It is therefore reported for coverage but excluded from the bold/underline ranking\.

### Experimental Setup

#### Optimization and training budget\.

The reportedCoCoS\-GPCrun trains from scratch for1010epochs with batch size128128while discarding incomplete final minibatches, yielding14,06014\{,\}060updates per epoch andT=140,600T=140\{,\}600updates in total\. We use AdamW with zero weight decay, mixed precision, and training seed4242\. The learning rate warms linearly for10,00010\{,\}000updates and then remains2×10−42\\times 10^\{\-4\}\. Training uses one RTX 4090\.

#### Partitioning, contrast\-set construction, and onset\.

The proposed configuration uses co\-expression\-guided gene partitioning and expression\-aware sampling\. Each minibatch processes two disjoint gene\-panel views with the same encoder parameters and sums their masked\-value MSE terms in one optimizer update\. Thus, the epoch and update budgets are not doubled, although the separate view and hard\-negative encodings are not FLOP\-matched to the single\-view reconstruction reference\. For each minibatch, we also sample one shared perturbation mask that independently reassigns each gene to the opposite view with probability5%5\\%\. We use a40%40\\%mask ratio, require at least5050observed genes per cell, and cap each view at512512genes\. The contrast set contains cross\-view and same\-view in\-batch negatives together withK=4K=4paired\-view, fixed\-identity value derangements\. The label\-free controller activates only after three consecutive probes show smoothed concordance above its calibrated threshold with a small recent slope, reconstruction error below its reference level, and repertoire above its reference level\. All onset hyperparameters and run\-specific calibration values are reported in the supplementary material\.

#### Frozen\-embedding evaluation\.

Under the unified ten\-dataset cell\-forward protocol, we extract each frozen whole\-cell embedding with maximum sequence length512512and evaluate it usingkk\-nearest neighbors \(k=10k\{=\}10\) and a learned linear probe on a fixed0\.7/0\.30\.7/0\.3train/test split\.

#### Comparators\.

We report scGPT\(Cuiet al\.[2024](https://arxiv.org/html/2608.00985#bib.bib3)\)under the same downstream protocol with an aligned gene vocabulary as an external reference\. We additionally report the public Geneformer V2\-104M checkpoint\(Theodoriset al\.[2023](https://arxiv.org/html/2608.00985#bib.bib4)\)\. After the common top\-512 biological\-gene selection, its frozen encoder receives the model\-specific median/rank encoding and outputs a 768\-dimensional cell embedding\. This evaluation matches the fixed cell rows, labels, seed\-42 split, L2 normalization,kk\-NN, and final\-epoch Linear@50 probes, but not native full\-transcriptome input, pretraining budget, or capacity\. Its Linear@50 value is also not numerically interchangeable with the archived legacy evaluator, so the row is excluded from the column rankings\. The Transformer implementation released with scBenchmark\(Qiet al\.[2025](https://arxiv.org/html/2608.00985#bib.bib1)\)is a full\-cell reconstruction\-only reference\. A split\-view reconstruction\-only control and a contrastive\-only control provide additional context\. The external references retain their respective pretraining architectures, batch sizes, and whole\-cell readout configurations\. Uniform sampling is the matched comparison for expression\-aware sampling\. Table[3](https://arxiv.org/html/2608.00985#Sx4.T3)additionally reports dedicated full\-cell MLM, split\-view MLM, balanced\-random panel, and no\-flip view controls\. Three fixed\-onset controls compare immediate contrast with one\- and two\-epoch reconstruction warmups within a shared training configuration\. The proposed GPC row reports the selected system configuration\. Historical runs that change multiple factors are reported separately as contextual evidence\.

### Does the Complete Method Learn Transferable Whole\-Cell Embeddings?

Under the unified ten\-dataset cell\-forward protocol,CoCoS\-GPCrecords mean accuracies of74\.82%74\.82\\%withkk\-NN \(Table[1](https://arxiv.org/html/2608.00985#Sx3.T1)\) and78\.85%78\.85\\%with a linear probe \(Table[2](https://arxiv.org/html/2608.00985#Sx4.T2)\), compared with70\.99%70\.99\\%and77\.02%77\.02\\%, respectively, for scGPT\. Among the ranked legacy rows, underkk\-NN,CoCoS\-GPCis strictly best on seven datasets and tied for best on dengue\. Within the same ranked set, under linear evaluation, it has the highest mean accuracy, ranks first on six datasets, and ranks second on three; Myeloid is the only dataset on which it is outside the top two\. The margin over scGPT is larger forkk\-NN \(\+3\.83\+3\.83points\) than for the linear probe \(\+1\.83\+1\.83points\), indicating that the measured gain is more pronounced in local neighborhood quality than in linear separability under these evaluators\. All values use training seed 42 and downstream split seed 42\.

For additional external\-model coverage, the daggered Geneformer V2\-104M row\(Theodoriset al\.[2023](https://arxiv.org/html/2608.00985#bib.bib4)\)records70\.38%70\.38\\%meankk\-NN accuracy \(Table[1](https://arxiv.org/html/2608.00985#Sx3.T1)\) and66\.22%66\.22\\%mean Linear@50 accuracy \(Table[2](https://arxiv.org/html/2608.00985#Sx4.T2)\)\. Under this post\-selection adaptation, its meankk\-NN score exceeds its mean final\-epoch linear score by7\.767\.76points\. We do not include the row in best/second\-best counts or use it for a leaderboard claim: its median/rank encoding consumes an adapted top\-512 view rather than native full\-transcriptome input, and its fixed Linear@50 evaluator is not numerically interchangeable with the archived legacy linear results\.

Table 3:Meankk\-NN and linear\-probe accuracies \(%\) for view\-design variants under the ten\-dataset cell\-forward protocol \(one training seed and one fixed downstream split\)\.
### How Do View\-Design Variants Compare?

Table[3](https://arxiv.org/html/2608.00985#Sx4.T3)compares six configurations\. The proposed expression\-aware variant records the highest mean point estimates \(74\.82%74\.82\\%kk\-NN and78\.85%78\.85\\%linear\), while balanced\-random panels remain close \(74\.34%74\.34\\%and78\.44%78\.44\\%\)\. Full\-cell and split\-view reconstruction are weaker\. The no\-flip run is also lower but never activates GPC, so it does not isolate the effect of panel\-assignment flips\.

The matched sampling pair changes only the within\-view sampler\. Expression\-aware sampling improves over uniform sampling by0\.880\.88kk\-NN and0\.250\.25linear points\. Its onset also shifts from96,00096\{,\}000to93,00093\{,\}000steps, so the comparison evaluates the sampler within the gated system rather than at a fixed onset\.

### Do Fixed\-Identity Negatives Mitigate the Gene\-Identity Shortcut?

Diagnostic\.If ordinary in\-batch negatives can be separated using gene\-set composition, the model should be highly confident against other\-cell candidates but less confident when a negative preserves the positive’s gene identities and changes only the gene–value correspondence\. We measure this behavior with the ordinary–identity confidence gap: ordinary cross\-view positive confidence minus confidence against four fixed\-identity value derangements\. A persistent positive gap indicates that identity\-matched candidates remain harder than ordinary negatives\.

Answer\.Starting from the same training state at step84,36084\{,\}360, matchedK=0K=0andK=4K=4branches run for1,0001\{,\}000updates and differ only in the number of fixed\-identity negatives\. At the final probe,K=0K=0is nearly certain against ordinary negatives \(0\.9970\.997confidence\) but is less confident against fixed\-identity derangements \(0\.6390\.639\), leaving a gap of0\.3580\.358\. WithK=4K=4, fixed\-identity confidence reaches0\.9690\.969and the gap narrows to0\.0280\.028\(Figure[3](https://arxiv.org/html/2608.00985#Sx4.F3)\)\. Within this controlled diagnostic, the collapsed gap indicates that fixed\-identity negatives mitigate reliance on gene\-set composition and promote sensitivity to gene–value correspondence\. This short fork does not establish downstream gains or training\-seed uncertainty\. We also do not compare raw training InfoNCE acrossKK, becauseK=4K=4adds four denominator logits\.

![Refer to caption](https://arxiv.org/html/2608.00985v1/x3.png)Figure 3:Fixed\-identity shortcut diagnostic\. From a shared checkpoint at step84,36084\{,\}360, matchedK=0K=0andK=4K=4branches run for1,0001\{,\}000updates and differ only in fixed\-identity negatives\. \(A\) Paired\-positive confidence against four value derangements\. \(B\) Confidence gap between ordinary cross\-view and fixed\-identity pools\. Both panels use fixed held\-out five\-way pools atτ=0\.07\\tau=0\.07; this short\-fork diagnostic does not estimate downstream performance or training\-seed uncertainty\.
### When Should Contrastive Learning Begin?

TheCoCoS\-GPCcontroller triggers at step93,00093\{,\}000, after all readiness conditions hold for three consecutive probes\. Before then, the contrastive weight is zero and neither contrastive gradients nor value\-derangement encodings are computed; afterward, the weight ramps up with the complete contrast set\.

Among the matched fixed\-onset controls in Table[4](https://arxiv.org/html/2608.00985#Sx4.T4), a two\-epoch reconstruction warmup performs best, improving over immediate contrast by1\.501\.50kk\-NN and1\.041\.04linear points\. The GPC row is a contextual reference rather than part of this matched comparison\.

Reevaluating the selected checkpoint over downstream split seeds 42, 43, and 44 gives74\.87±0\.06%74\.87\\pm 0\.06\\%meankk\-NN and78\.77±0\.07%78\.77\\pm 0\.07\\%mean linear accuracy\.

Table 4:Contrastive\-onset comparison under the ten\-dataset cell\-forward protocol \(training and downstream split seed 42\)\. Onset is in thousands of optimizer steps\. The fixed\-onset controls differ only in onset; GPC is the proposed configuration shown as an unmatched reference\.
### Do the Embeddings Support Gene\-Regulatory Edge Prediction?

We evaluate frozen gene embeddings for gene\-regulatory edge prediction\. Each candidate TF–target pair is represented by the two embeddings, their element\-wise product, cosine similarity, and Euclidean distance, and classified using logistic\-regression and two\-layer MLP probes\. Because the task is class\-imbalanced, we report both AUROC and AUPRC, with AUPRC more directly reflecting performance on the sparse positive\-edge class\.

Table[5](https://arxiv.org/html/2608.00985#Sx4.T5)reports results on six Specific 1000\-gene networks under the full\-candidate negative protocol\.CoCoS\-GPCrecords the highest mean point estimates among the compared variants for AUROC \(0\.8733\) and AUPRC \(0\.7120\)\. Winners vary across networks, so these averages indicate competitive biological transfer rather than uniform or statistically significant superiority\.

Table 5:Mean AUROC/AUPRC point estimates for GRN edge prediction on six BEELINE Specific 1000\-gene networks with full\-candidate negatives, averaging logistic\-regression and MLP probes\. Bold marks the highest mean\.

## Conclusion

CoCoSaddresses the mismatch between gene\-level reconstruction and whole\-cell representation learning by jointly structuring complementary gene views, contrast sets with identity\-matched value negatives, and data\-dependent contrastive onset\. Under the unified ten\-dataset cell\-forward protocol, the resultingCoCoS\-GPCembeddings attain the highest meankk\-NN accuracy among the compared methods, expression\-aware sampling yields higher point estimates than its matched uniform counterpart, and the held\-out controller activates contrastive learning without downstream labels\. The model’s frozen gene embeddings yield the highest mean AUROC and AUPRC point estimates among the compared variants in the six\-network GRN evaluation, although the variant with the highest point estimate differs across individual networks\. Together, these results support complementary\-view contrastive learning as a useful extension to masked expression reconstruction across cell\-type annotation and gene\-regulatory edge prediction, within the evaluated datasets and protocols\.

## Acknowledgments

This work was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences under Grant No\. XDA0460205\.

## References

- ScConcept: contrastive pretraining for technology\-agnostic single\-cell representations beyond reconstruction\.bioRxiv\.External Links:[Document](https://dx.doi.org/10.1101/2025.10.14.682419),[Link](http://dx.doi.org/10.1101/2025.10.14.682419)Cited by:[Single\-cell representation learning\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px2.p1.1)\.
- Y\. Bengio, J\. Louradour, R\. Collobert, and J\. Weston \(2009\)Curriculum learning\.InProceedings of the 26th Annual International Conference on Machine Learning,pp\. 41–48\.External Links:[Document](https://dx.doi.org/10.1145/1553374.1553380),[Link](https://doi.org/10.1145/1553374.1553380)Cited by:[Adaptive training\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px3.p1.1)\.
- J\. Chen, Z\. Hu, S\. Sun, Q\. Tan, Y\. Wang, Q\. Yu, L\. Zong, L\. Hong, J\. Xiao, T\. Shen, I\. King, and Y\. Li \(2022\)Interpretable RNA foundation model from unannotated data for highly accurate RNA structure and function predictions\.External Links:2204\.00300,[Link](https://arxiv.org/abs/2204.00300)Cited by:[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1)\.
- T\. Chen, S\. Kornblith, M\. Norouzi, and G\. E\. Hinton \(2020\)A simple framework for contrastive learning of visual representations\.InProceedings of the 37th International Conference on Machine Learning, ICML 2020,Proceedings of Machine Learning Research, Vol\.119,pp\. 1597–1607\.External Links:[Link](http://proceedings.mlr.press/v119/chen20j.html)Cited by:[Introduction](https://arxiv.org/html/2608.00985#Sx1.p2.1)\.
- Z\. Chen, V\. Badrinarayanan, C\. Lee, and A\. Rabinovich \(2018\)GradNorm: gradient normalization for adaptive loss balancing in deep multitask networks\.InProceedings of the 35th International Conference on Machine Learning, ICML 2018,Proceedings of Machine Learning Research, Vol\.80,pp\. 794–803\.External Links:[Link](https://proceedings.mlr.press/v80/chen18a.html)Cited by:[Adaptive training\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px3.p1.1)\.
- M\. Ciortan and M\. Defrance \(2021\)Contrastive self\-supervised clustering of scrna\-seq data\.BMC Bioinformatics22\(1\),pp\. 280\.External Links:[Document](https://dx.doi.org/10.1186/s12859-021-04210-8),[Link](https://doi.org/10.1186/s12859-021-04210-8)Cited by:[Single\-cell representation learning\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px2.p1.1)\.
- H\. Cui, C\. Wang, H\. Maan, K\. Pang, F\. Luo, N\. Duan, and B\. Wang \(2024\)ScGPT: toward building a foundation model for single\-cell multi\-omics using generative ai\.Nature Methods21,pp\. 1470–1480\.External Links:[Document](https://dx.doi.org/10.1038/s41592-024-02201-0)Cited by:[Introduction](https://arxiv.org/html/2608.00985#Sx1.p1.1),[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.00985#Sx3.T1.1.4.3.1),[Comparators\.](https://arxiv.org/html/2608.00985#Sx4.SSx2.SSS0.Px4.p1.1),[Table 2](https://arxiv.org/html/2608.00985#Sx4.T2.1.4.3.1)\.
- H\. Dalla\-Torre, L\. Gonzalez, J\. Mendoza\-Revilla, N\. Lopez Carranza, A\. H\. Grzywaczewski, F\. Oteri, C\. Dallago, E\. Trop, B\. P\. de Almeida, H\. Sirelkhatim, G\. Richard, M\. Skwark, K\. Beguir, M\. Lopez, and T\. Pierrot \(2025\)Nucleotide transformer: building and evaluating robust foundation models for human genomics\.Nature Methods22\(2\),pp\. 287–297\.External Links:[Document](https://dx.doi.org/10.1038/s41592-024-02523-z),[Link](https://doi.org/10.1038/s41592-024-02523-z)Cited by:[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1)\.
- J\. Ding, J\. Lin, S\. Jiang, Y\. Wang, Z\. Miao, Z\. Fang, J\. Tang, M\. Li, and X\. Qiu \(2025\)Tabula: a tabular self\-supervised foundation model for single\-cell transcriptomics\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2025/hash/95d590995a8722259c61e094b62b25ac-Abstract-Conference.html)Cited by:[Single\-cell representation learning\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px2.p1.1)\.
- W\. Han, Y\. Cheng, J\. Chen, H\. Zhong, Z\. Hu, S\. Chen, L\. Zong, L\. Hong, T\. Chan, I\. King, X\. Gao, and Y\. Li \(2022\)Self\-supervised contrastive learning for integrative single cell RNA\-seq data analysis\.Briefings in Bioinformatics23\(5\),pp\. bbac377\.External Links:[Document](https://dx.doi.org/10.1093/bib/bbac377),[Link](https://doi.org/10.1093/bib/bbac377)Cited by:[Single\-cell representation learning\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px2.p1.1)\.
- M\. Hao, J\. Gong, X\. Zeng, C\. Liu, Y\. Guo, X\. Cheng, T\. Wang, J\. Ma, X\. Zhang, and L\. Song \(2024\)Large\-scale foundation model on single\-cell transcriptomics\.Nature Methods21,pp\. 1481–1491\.External Links:[Document](https://dx.doi.org/10.1038/s41592-024-02305-7)Cited by:[Introduction](https://arxiv.org/html/2608.00985#Sx1.p1.1),[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1)\.
- Y\. Ji, Z\. Zhou, H\. Liu, and R\. V\. Davuluri \(2021\)DNABERT: pre\-trained bidirectional encoder representations from transformers model for DNA\-language in genome\.Bioinformatics37\(15\),pp\. 2112–2120\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/btab083),[Link](https://doi.org/10.1093/bioinformatics/btab083)Cited by:[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1)\.
- Y\. Kong, L\. Liu, J\. Wang, and D\. Tao \(2021\)Adaptive curriculum learning\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 5067–5076\.External Links:[Link](https://openaccess.thecvf.com/content/ICCV2021/html/Kong_Adaptive_Curriculum_Learning_ICCV_2021_paper.html)Cited by:[Adaptive training\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px3.p1.1)\.
- D\. Kotliar, A\. Veres, M\. A\. Nagy, S\. Tabrizi, E\. Hodis, D\. A\. Melton, and P\. C\. Sabeti \(2019\)Identifying gene expression programs of cell\-type identity and cellular activity with single\-cell RNA\-Seq\.eLife8,pp\. e43803\.External Links:[Document](https://dx.doi.org/10.7554/eLife.43803),[Link](https://doi.org/10.7554/eLife.43803)Cited by:[Introduction](https://arxiv.org/html/2608.00985#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.00985#Sx1.p2.1),[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1)\.
- S\. Liu, S\. James, A\. J\. Davison, and E\. Johns \(2022\)Auto\-Lambda: disentangling dynamic task relationships\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=KKeCMim5VN)Cited by:[Adaptive training\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px3.p1.1)\.
- W\. Long, H\. Su, J\. Xiong, and Y\. Zhang \(2025\)MutBERT: probabilistic genome representation improves genomics foundation models\.Bioinformatics41\(Supplement 1\),pp\. i294–i303\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/btaf229),[Link](https://doi.org/10.1093/bioinformatics/btaf229)Cited by:[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1)\.
- E\. A\. Platanios, O\. Stretcu, G\. Neubig, B\. Poczos, and T\. Mitchell \(2019\)Competence\-based curriculum learning for neural machine translation\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 1162–1172\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1119),[Link](https://aclanthology.org/N19-1119/)Cited by:[Adaptive training\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px3.p1.1)\.
- A\. Pratapa, A\. P\. Jalihal, J\. N\. Law, A\. Bharadwaj, and T\. M\. Murali \(2020\)Benchmarking algorithms for gene regulatory network inference from single\-cell transcriptomic data\.Nature Methods17\(2\),pp\. 147–154\.External Links:[Document](https://dx.doi.org/10.1038/s41592-019-0690-6),[Link](https://doi.org/10.1038/s41592-019-0690-6)Cited by:[Downstream evaluation\.](https://arxiv.org/html/2608.00985#Sx4.SSx1.SSS0.Px2.p1.1)\.
- J\. Qi, Y\. Cui, K\. Guo, X\. Zhang, J\. Huang, and G\. Xie \(2025\)A simple and comprehensive benchmark for single\-cell transcriptomics\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 676–684\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i1.32049)Cited by:[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1),[Backbone and Masked Pretraining](https://arxiv.org/html/2608.00985#Sx3.SSx1.p2.3),[Table 1](https://arxiv.org/html/2608.00985#Sx3.T1.1.3.2.1),[Pretraining corpus\.](https://arxiv.org/html/2608.00985#Sx4.SSx1.SSS0.Px1.p1.5),[Downstream evaluation\.](https://arxiv.org/html/2608.00985#Sx4.SSx1.SSS0.Px2.p1.1),[Comparators\.](https://arxiv.org/html/2608.00985#Sx4.SSx2.SSS0.Px4.p1.1),[Table 2](https://arxiv.org/html/2608.00985#Sx4.T2.1.3.2.1),[Table 5](https://arxiv.org/html/2608.00985#Sx4.T5.1.2.1.1)\.
- C\. V\. Theodoris, L\. Xiao, A\. Chopra, M\. D\. Chaffin, Z\. R\. Al Sayed, M\. C\. Hill, H\. Mantineo, E\. M\. Brydon, Z\. Zeng, X\. S\. Liu, and P\. T\. Ellinor \(2023\)Transfer learning enables predictions in network biology\.Nature618,pp\. 616–624\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06139-9)Cited by:[Introduction](https://arxiv.org/html/2608.00985#Sx1.p1.1),[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.00985#Sx3.T1.1.1.1),[Downstream evaluation\.](https://arxiv.org/html/2608.00985#Sx4.SSx1.SSS0.Px2.p1.1),[Comparators\.](https://arxiv.org/html/2608.00985#Sx4.SSx2.SSS0.Px4.p1.1),[Does the Complete Method Learn Transferable Whole\-Cell Embeddings?](https://arxiv.org/html/2608.00985#Sx4.SSx3.p2.5),[Table 2](https://arxiv.org/html/2608.00985#Sx4.T2.1.1.1)\.
- J\. Xiong, N\. Yin, S\. Liang, H\. Li, Y\. Wang, D\. Ai, and J\. Wang \(2025\)Cross\-attention graph neural networks for inferring gene regulatory networks with skewed degree distribution\.BMC Bioinformatics26\(1\),pp\. 179\.External Links:[Document](https://dx.doi.org/10.1186/s12859-025-06186-1),[Link](https://doi.org/10.1186/s12859-025-06186-1)Cited by:[Introduction](https://arxiv.org/html/2608.00985#Sx1.p1.1)\.
- F\. Yang, W\. Wang, F\. Wang, Y\. Fang, D\. Tang, J\. Huang, H\. Lu, and J\. Yao \(2022a\)ScBERT as a large\-scale pretrained deep language model for cell type annotation of single\-cell rna\-seq data\.Nature Machine Intelligence4,pp\. 852–866\.External Links:[Document](https://dx.doi.org/10.1038/s42256-022-00534-z)Cited by:[Introduction](https://arxiv.org/html/2608.00985#Sx1.p1.1),[Single\-cell foundation models\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px1.p1.1)\.
- M\. Yang, Y\. Yang, C\. Xie, M\. Ni, J\. Liu, H\. Yang, F\. Mu, and J\. Wang \(2022b\)Contrastive learning enables rapid mapping to multimodal single\-cell atlas of multimillion scale\.Nature Machine Intelligence4,pp\. 696–709\.External Links:[Document](https://dx.doi.org/10.1038/s42256-022-00518-z),[Link](https://doi.org/10.1038/s42256-022-00518-z)Cited by:[Single\-cell representation learning\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px2.p1.1)\.
- S\. Zhao, J\. Zhang, Y\. Wu, Y\. Luo, and Z\. Nie \(2024\)LangCell: language\-cell pre\-training for cell identity understanding\.External Links:2405\.06708,[Link](https://arxiv.org/abs/2405.06708)Cited by:[Single\-cell representation learning\.](https://arxiv.org/html/2608.00985#Sx2.SS0.SSS0.Px2.p1.1)\.