Separating Representation from Reconstruction Enables Scalable Text Encoders

# Separating Representation from Reconstruction Enables Scalable Text Encoders
Source: [https://arxiv.org/html/2607.04011](https://arxiv.org/html/2607.04011)
###### Abstract

While decoders have rapidly scaled, encoders have remained largely unchanged since BERT\. We revisit this disparity by frozen backbone evaluation via probing\. Under this lens, the representations of BERT encoders become increasinglyunexploitableby frozen probes, despite improved perplexity\. The misalignment originates in BERT’s flat design, which couples representation learning to the token reconstruction loss\. We proposeCrossBERT, a two\-part architecture that separates the learning of high\-quality encoded representations from the rigid grounding of token reconstruction\. This design further enables high masking ratios \(≥50%\\geq 50\\%\) and gradient collection over all tokens via aComplementary Masking Strategy, respectively increasing throughput by1\.51\.5to2×2\\timesand sample efficiency by2×2\\times\. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB\(eng, v2\) and frozen GLUE benchmarks\.

BERT, encoder, SSL, text, ICML

![Refer to caption](https://arxiv.org/html/2607.04011v1/x1.png)

![Refer to caption](https://arxiv.org/html/2607.04011v1/x2.png)

Figure 1:Top\.Frozen evaluation of encoders on GLUE, linear and KNN probes are fitted on the average representation of a frozen backbone\.Bottom\.MTEB\(eng, v2\) score\. ‘\(frozen\)’ means only the pooler is finetuned on top of the frozen features of the encoder i\.e frozen backbone\. ‘\(unfrozen\)’ means that the full network \(including the backbone\) is finetuned end\-to\-end\. Both are finetuned only on MS\-MARCO for one epoch with hard\-negatives\.## 1Introduction

Encoders are critical for a variety of modern applications, ranging from large\-scale data curation and retrieval\-augmented generation to recommendation systems\. However, encoder architectures have remained largely unchallenged since BERT, most improvements stem from scaling training datasets\. The research community has focused on elaborate post\-training pipelines utilizing pre\-trained models merely as initialization\(Wanget al\.,[2022](https://arxiv.org/html/2607.04011#bib.bib18)\)\. This heavy reliance on downstream finetuning does not show the shortcomings of the pre\-trained backbone, hindering its development\.Dervishiet al\.\([2025](https://arxiv.org/html/2607.04011#bib.bib1)\)recently demonstrated the cost of neglecting pre\-training: frontier pre\-trained backbones like ModernBERT\(Warneret al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib2)\)and NeoBERT\(Bretonet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib3)\)are vastly overtrained relative to their size\.

![Refer to caption](https://arxiv.org/html/2607.04011v1/x3.png)Figure 2:Left\.The Complementary Masking Strategy \(CMS\) augments a batch of tokens into two complementary masked views by replacing tokens with<MASK\>\. Masked tokens are in red; the unmasked tokens of the two views are in green and blue\. Numbers indicate positional indices\.Right\.CrossBERT predicts one view \(green\) from the other \(blue\), and vice\-versa\. Both views are processed in parallel with an attention mask isolating one view from the other, preventing information leakage\. The encoder takes the unmasked tokens and their positional indices as input\. The lightweight predictor takes<MASK\>placeholders and their positional indices; it never sees the masked tokens’ content\. The predictor attends to the encoder’s output representations via cross\-attention and is trained to reconstruct the complementary view with a Cross\-Entropy \(CE\) loss\. Solid color boxes denote tokens \(red<MASK\>, blue/green true values\); hatched boxes denote representations \(blue from the encoder, pink from the predictor\)\.Concurrently, the vision encoder community has continued to improve pre\-trained backbones via scaling strategies\(Bolyaet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib23); Sunet al\.,[2023](https://arxiv.org/html/2607.04011#bib.bib24); Oquabet al\.,[2023](https://arxiv.org/html/2607.04011#bib.bib11); Siméoniet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib12)\)and novel training recipes\(Fuet al\.,[2024](https://arxiv.org/html/2607.04011#bib.bib7); Chenet al\.,[2020](https://arxiv.org/html/2607.04011#bib.bib9); Caronet al\.,[2021](https://arxiv.org/html/2607.04011#bib.bib10); Darcetet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib8); Assranet al\.,[2023](https://arxiv.org/html/2607.04011#bib.bib19); Bardeset al\.,[2021](https://arxiv.org/html/2607.04011#bib.bib20); Garridoet al\.,[2024](https://arxiv.org/html/2607.04011#bib.bib21); Heet al\.,[2022](https://arxiv.org/html/2607.04011#bib.bib25)\)\. Since these encoders are often utilized frozen, i\.e\. without modifications, evaluation standards have naturally prioritized frozen benchmarks\. Therefore encouraging pre\-training innovations\. Effective representation learning is defined by the interpretability and versatility of frozen embeddings\.

Motivated by recent architectural insights in the vision community, we revisit the design and evaluation of text encoders\. Specifically, we conduct a comprehensive scaling analysis, evaluating the representations of frozen pretrained backbones alongside standard finetuning protocols\. This shift in perspective reveals a counterintuitive phenomenon: as modern BERT models scale, their features become increasingly unexploitable by frozen probes\. To resolve this bottleneck, we introduce CrossBERT\. Inspired byHeet al\.\([2022](https://arxiv.org/html/2607.04011#bib.bib25)\); Fuet al\.\([2024](https://arxiv.org/html/2607.04011#bib.bib7)\), CrossBERT decouples representation learning from token reconstruction by appending a lightweight cross\-attention predictor to the final backbone layers\. This architectural shift ensures that the backbone focuses entirely on feature extraction, while token reconstruction is isolated within the predictor\. We demonstrate that this novel design preserves representation quality as the model scales\. Furthermore, even when evaluated in a data\-constrained contrastive setup using only MS\-MARCO, CrossBERT mitigates saturation and exhibits a significantly superior scaling trends compared to standard baselines\.

Throughout the paper, our baseline BERT is a modern implementation \(see[Section5\.1](https://arxiv.org/html/2607.04011#S5.SS1)\)\. BERT and CrossBERT differ only on the architecture structure: flat vs bipartite\. In[Section2](https://arxiv.org/html/2607.04011#S2)we explain our conjecture for the failure mode of the BERT architecture and how CrossBERT solves it\. In[Section3](https://arxiv.org/html/2607.04011#S3)we detail the architecture of CrossBERT and present its advantages compared to the usual BERT models\. In Section[4](https://arxiv.org/html/2607.04011#S4)we summarize the new frozen evaluation methods that we use to measure solely the pre\-training performance\. Finally we present our experimental results and discussion in Section[5](https://arxiv.org/html/2607.04011#S5)and conclude with future work in Section[8](https://arxiv.org/html/2607.04011#S8)\.

Contributions

1. C1\.The CrossBERT Architecture\.We introduce a bipartite encoder that ensures consistent performance scaling on frozen evaluations\. To the best of our knowledge, this is the first Masked Autoencoder for text\.
2. C2\.High\-Efficiency Training via High Masking Ratio\.We demonstrate CrossBERT’s robustness at masking ratios\>50%\>50\\%, accelerating training throughput by≈1\.5–2×\\approx 1\.5\\text\{\-\-\}2\\times\. Furthermore, this tolerance enables aComplementary Masking Strategy\(CMS\), which processes the inverse mask in parallel\. Effectively doubling the sample efficiency by collecting gradients from every token in the sequence\.
3. C3\.Scaling Laws under Frozen Evaluation\.We conduct a scaling analysis \(2×10182\\times 10^\{18\}to1×10211\\times 10^\{21\}FLOPs\) to quantify intrinsic representation quality\. This exposes a performance gap in standard BERTs and validates CrossBERT’s superior extractability, an advantage that we show extends to MTEB tasks\.

## 2CrossBERT: Intuition

#### Problem\.

BERT representations worsen as we scale compute as can be seen in Fig\.[1](https://arxiv.org/html/2607.04011#S0.F1)\.

#### Explanation\.

We conjecture that the reason behind such degradation stems from the Masked Language Modeling \(MLM\) objective\(Devlinet al\.,[2019](https://arxiv.org/html/2607.04011#bib.bib5)\)applied on the flat design of the BERT architecture\.

MLM trains an encoder to reconstruct a corrupted input sequence\. Given a sequence of tokensX=\{x1,…,xN\}X=\\\{x\_\{1\},\\dots,x\_\{N\}\\\}, a subset of indicesℳ\\mathcal\{M\}is selected for masking\. The tokens at these positions are replaced by a special token<MASK\>, yielding a corrupted sequenceX~\\tilde\{X\}\. The model processesX~\\tilde\{X\}to generate contextualized representations, and the objective is to minimize the negative log\-likelihood of the original tokensxmx\_\{m\}at the masked positions:

ℒMLM=−∑m∈ℳlog⁡P​\(xm∣X~\)\\mathcal\{L\}\_\{\\text\{MLM\}\}=\-\\sum\_\{m\\in\\mathcal\{M\}\}\\log P\(x\_\{m\}\\mid\\tilde\{X\}\)\(1\)
whereP​\(xm∣X~\)P\(x\_\{m\}\\mid\\tilde\{X\}\)denotes the probability assigned to the true tokenxmx\_\{m\}by the prediction head\.Hence the objective only measures the token reconstruction ability of the model but not the actual quality of its representations\.

Since the “flat” design of the BERT architecture does not explicitly separate representation creation from token reconstruction, the representations remain overly “grounded” in the local signal required to predict missing tokens instead of being versatile high\-level abstractions\. Some evidence of this phenomenon is displayed in[Table1](https://arxiv.org/html/2607.04011#S3.T1), where the BERT encoder is probed at different depths\. Notably, retrieving representations from earlier layers, rather than the final output, yields slightly improved performance, showcasing the over\-specialization of these last layers\.

#### Solution\.

Inspired by Masked Auto\-Encoders \(MAE\) approaches in vision\(Heet al\.,[2022](https://arxiv.org/html/2607.04011#bib.bib25); Fuet al\.,[2024](https://arxiv.org/html/2607.04011#bib.bib7)\), we propose CrossBERT: a bipartite architecture that separates the heavy lifting of representation creation \(Encoder\) from the specific task of token reconstruction \(Predictor\), as illustrated in Fig\.[2](https://arxiv.org/html/2607.04011#S1.F2)\.

## 3CrossBERT: Architecture & Advantages

### 3\.1Architecture

A sketch of the architecture is shown in the right panel of[Figure2](https://arxiv.org/html/2607.04011#S1.F2)\. The input sequence is partitioned into a visible set \(processed by the encoder\) and a masked set \(processed by the predictor\)\. To preserve sequence order, we encode the position of each token through RoPE, ensuring the encoder and predictor are aware of which positions are missing\. The predictor is a few transformer blocks thatonlycross\-attend to the encoder representations\. By removing self\-attention between masked tokens, we force the predictor to act strictly as a “readout” interface that must satisfy its objective solely by querying the encoder’s embeddings\. Additionaly we implement modern architectural optimizations\(Warneret al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib2); Bretonet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib3)\)such as RMSNorm\.

To set the size of the encoder and predictor we align with MAE\(Heet al\.,[2022](https://arxiv.org/html/2607.04011#bib.bib25)\)\. The predictor shares the encoder’s hidden dimension, but is significantly shallower \(approximately one\-forth of the encoder’s depth\)\. Ablation on the predictor’s shape and size can be found in[AppendixA](https://arxiv.org/html/2607.04011#A1)\. We gain two insights from this ablation\. First, the specific aspect ratio \(width vs\. depth\) of the predictor is not significant\. Second, while increasing the predictor’s capacity can yield some performance gains \(with diminishing returns\), it comes at the cost of slower training\. Since our objective is to obtain rich frozen features, we decide to allocate most of the compute to the encoder\.

Table 1:Layer\-wise Frozen GLUE Score Analysis on BERT\. The scores are obtained by fitting a linear probe on the frozen features as explained in[Section4\.1](https://arxiv.org/html/2607.04011#S4.SS1)Modellastlast\-1last\-2last\-32020thAvg\. Improv\.BERT 239M67\.468\.569\.570\.268\.7\+1\.8BERT 1\.21B64\.665\.865\.065\.966\.6\+1\.2
### 3\.2Emerging Advantages

This design choice leads to several advantages: predictor’s transfer learning; robustness to higher masking ratios; the ability to use the Complementary Masking Strategy \(CMS\); better data efficiency and lower training costs\.

#### Transferability\.

The predictor learns to extract information only from the encoder representation during pre\-training\. Hence, the predictor functions as a learned pooling mechanism\. It can be effectively re\-used as a warm\-started module for downstream finetuning, serving as an efficient bridge between the frozen encoder features and the target task\.

#### Robustness to higher masking ratios\.

BERT architectures typically suffer performance degradation when masking ratios exceed20​–​40%20\\text\{\-\-\}40\\%\(Wettiget al\.,[2023](https://arxiv.org/html/2607.04011#bib.bib38)\)\(see[AppendixC](https://arxiv.org/html/2607.04011#A3)\)\. We challenge this limitation by profiling the frozen representation quality of CrossBERT across multiple masking ratios\. As shown in[Table2](https://arxiv.org/html/2607.04011#S3.T2), CrossBERT exhibits remarkable stability: increasing masking from20%20\\%to50%50\\%incurs a negligible drop in GLUE performance \(−0\.7%\-0\.7\\%\)\. This resilience confirms that our bipartite architecture successfully insulates representation learning from the difficulty of the reconstruction task\. Increasing to higher masking ratios directly translates into reduced training costs, which opens the door to aComplementary Masking Strategy \(CMS\)\.

Table 2:Average Frozen GLUE score of CrossBERT across different masking ratios keeping the same data budget\.Masking ratio \(%\)20405065CrossBERT73\.873\.773\.172\.4
#### CMS\.

Complementary Masking Strategy is depicted in the left panel of[Figure2](https://arxiv.org/html/2607.04011#S1.F2)\. Every batch is augmented with its inverse mask, i\.e\. creating two complementary views of the original token sequence\. This allows the model to predict and learn from every token in the sequence\. The model ensures that information from the original sequence does not “leak” into the complementary view within the same batch by applying a unique attention mask on each view\. Hence, both views can be processed simultaneously\. For BERT, this approach does not mix well with the 20% – 40 % masking ratio requirement as the inverse view would land on the 60% – 80% ratio\. Because CrossBERT remains effective beyond50%50\\%masking, it can learn from both the original and the complementary sequence efficiently\.

#### Sample efficiency\.

MLM is known for being sample\-inefficient; the model learns only from a fraction of tokens per pass since gradients are only computed on masked tokens\. However by utilizing CMS, CrossBERT processes the visible tokens and its inverse, effectively seeing all the tokens in the sequence and predicting all the tokens from the predictor in one go\. Empirically, our results \([AppendixB](https://arxiv.org/html/2607.04011#A2)\) confirm that CMS does not damage the performance of the baseline compared to standard masking\. Instead, CMS effectively doubles sample efficiency, requiring half the training data to reach the same performances\.

#### Computational efficiency\.

Even if BERT were to employ CMS, the computational cost would be prohibitively expensive: two forward\-backward passes on the full sequence\. On the other hand, the encoder of CrossBERT drops masked tokens\. Hence the cost of training with CMS is just one forward\-backward pass of the encoder and predictor on the full sequence\.

Table 3:Results on GLUE and MTEB\(eng, v1\) under both full\-finetuning \(unfrozen backbone\) and frozen probing \([Sections4\.1](https://arxiv.org/html/2607.04011#S4.SS1)and[4\.2](https://arxiv.org/html/2607.04011#S4.SS2)\)\. We report score for MTEB\(eng, v1\) to be able to compare with previous work\. See[AppendixE](https://arxiv.org/html/2607.04011#A5)for the scores on MTEB\(eng,v2\)\.GLUEModelParamsFLOPStpsMNLIQNLIQQPRTESSTMRPCCoLASTSAvg\.BERT239M6\.8e19123k86\.590\.088\.585\.395\.191\.963\.091\.286\.4CrossBERT279M4\.1e19207k86\.490\.388\.485\.194\.891\.465\.591\.586\.6Electra258M7\.5e19116k88\.793\.389\.183\.494\.692\.071\.090\.087\.8ModernBERT352M4\.9e21\-90\.895\.292\.792\.197\.191\.771\.492\.890\.5NeoBERT198M2\.9e21\-88\.993\.990\.791\.095\.893\.464\.892\.188\.8OptiBERT239M7\.0e19\-86\.692\.190\.383\.292\.691\.059\.690\.885\.8DeBERTaV3304M\>\>2e20\-91\.996\.093\.092\.796\.991\.975\.393\.091\.4GLUE Linear probeBERT239M6\.8e19123k58\.674\.877\.759\.984\.976\.727\.479\.567\.4CrossBERT279M4\.1e19207k61\.681\.881\.464\.390\.677\.947\.984\.873\.8Electra258M7\.5e19116k69\.283\.682\.766\.886\.979\.658\.887\.576\.9ModernBERT352M4\.9e21\-63\.780\.080\.761\.086\.774\.040\.783\.071\.2NeoBERT198M2\.9e21\-47\.969\.872\.956\.078\.069\.110\.166\.158\.7DeBERTaV3304M\>\>2e20\-76\.686\.985\.875\.886\.581\.967\.190\.181\.3MTEB\(eng, v1\) Full Contrastive Finetuning on MS\-MARCOClass\.Clust\.PairClass\.Rerank\.Retriev\.STSSumm\.Avg\.OverallBERT239M6\.8e19123k62\.735\.180\.751\.543\.075\.829\.854\.153\.9CrossBERT279M4\.1e19207k67\.631\.580\.552\.242\.575\.331\.654\.554\.1Electra258M7\.5e19116k62\.127\.777\.948\.834\.372\.429\.150\.349\.0ModernBERT∗352M4\.9e21\-62\.438\.765\.550\.123\.168\.327\.846\.9\-NeoBERT∗198M2\.9e21\-61\.640\.876\.251\.231\.674\.830\.751\.3\-OptiBERT×239M7\.0e19\-67\.536\.175\.848\.123\.379\.730\.051\.5\-DeBERTaV3∗304M\>\>2e20\-45\.916\.445\.040\.84\.040\.129\.927\.1\-∗Full finetuning on much larger dataset including MSMARCO, StackOverFlowDupQuestion, Fever, STS12, and STSBenchmark and AllNLI\.×Full finetuning on AllNLI only\.MTEB\(eng, v1\) Contrastive Finetuning over frozen features on MS\-MARCOBERT239M6\.8e19123k60\.129\.764\.643\.719\.262\.730\.544\.342\.1CrossBERT279M4\.1e19207k65\.737\.178\.051\.440\.772\.330\.753\.753\.6Electra258M7\.5e19116k41\.312\.835\.432\.40\.032\.826\.926\.222\.4

## 4Method

In this section we aim to describe how we prove that CrossBERT works better than the current BERT recipe\.

Historically, encoder evaluations have relied heavily on full finetuning for downstream tasks ranging from classification to information retrieval\. Doing so, makes it hard to judge whether the final downstream performance is a result of finetuning or original pretrained representation quality\. Therefore, we evaluate performance by freezing the encoder, which allows us to directly measure the pretrained representation quality\.

We focus on two evaluation benchmarks\. While GLUE\(Wanget al\.,[2018](https://arxiv.org/html/2607.04011#bib.bib15)\)served as the gold standard for early pre\-training, the field, especially within the contrastive finetuning landscape, has since adopted the MTEB benchmark\(Muennighoffet al\.,[2023](https://arxiv.org/html/2607.04011#bib.bib6)\)to better assess embedding quality\. We now describe how to adapt these established benchmarks to a frozen evaluation protocol\.

### 4\.1GLUE frozen evaluation

Given the historical importance of classification in encoder evaluation, we use classification tasks to probe feature quality throughout the training process\. Specifically, the output of the last layer is averaged across all tokens to generate a single representation of the sequence\. A classifier is then optimized on these frozen features\. To ensure efficiency, these probes are fitted in parallel, necessitating only a single forward pass over the training set\. Two distinct types of classifiers are used:

#### Linear probing\.

The linear probes are optimized via Ridge Regression, which minimizes the standard least\-squares error augmented by aL2L\_\{2\}penalty term\. The loss function is defined as:

ℒRidge​\(𝐖\)=‖𝐘−𝐗𝐖‖F2\+λ​‖𝐖‖F2\\mathcal\{L\}\_\{\\text\{Ridge\}\}\(\\mathbf\{W\}\)=\\\|\\mathbf\{Y\}\-\\mathbf\{X\}\\mathbf\{W\}\\\|\_\{F\}^\{2\}\+\\lambda\\\|\\mathbf\{W\}\\\|\_\{F\}^\{2\}\(2\)where𝐗\\mathbf\{X\}represents the frozen encoder representations,𝐘\\mathbf\{Y\}the target labels, andλ\\lambdathe regularization coefficient controlling the penalty strength\. The optimal weights𝐖∗\\mathbf\{W\}^\{\*\}are computed directly via the closed\-form analytical solution:

𝐖∗=\(𝐗⊤​𝐗\+λ​𝐈\)−1​𝐗⊤​𝐘\\mathbf\{W\}^\{\*\}=\(\\mathbf\{X\}^\{\\top\}\\mathbf\{X\}\+\\lambda\\mathbf\{I\}\)^\{\-1\}\\mathbf\{X\}^\{\\top\}\\mathbf\{Y\}\(3\)This approach allows for rapid, deterministic fitting across multiple regularization strengths without the need for iterative optimization\. Moreover, this procedure is done on GPU leveragingtorch\.linalg\.solvemaking it even faster\. In practice, we sweep multiple logspacedλ\\lambdaas we notice that some dataset, and especially small ones, are very sensitive to this hyperparameter\.

#### kNN probing\.

As a non\-parametric complement to linear evaluation, k\-Nearest Neighbors \(kNN\) assess the intrinsic geometry of the representation space\. For a given queryzz, the prediction is determined by a majority vote among the set of itskkclosest neighbors𝒩k​\(z\)\\mathcal\{N\}\_\{k\}\(z\), identified by minimizing the distance metricdd\(eitherL2L\_\{2\}or Cosine\):

𝒩k​\(z\)\\displaystyle\\mathcal\{N\}\_\{k\}\(z\)=arg​min𝒮⊂𝒟,\|𝒮\|=k​∑xj∈𝒮d​\(z,xj\)\\displaystyle=\\underset\{\\mathcal\{S\}\\subset\\mathcal\{D\},\|\\mathcal\{S\}\|=k\}\{\\operatorname\{arg\\,min\}\}\\sum\_\{x\_\{j\}\\in\\mathcal\{S\}\}d\(z,x\_\{j\}\)\(4\)y^\\displaystyle\\hat\{y\}=arg​maxc∈𝒞​∑xi∈𝒩k​\(z\)𝟙​\(yi=c\)\\displaystyle=\\operatorname\*\{arg\\,max\}\_\{c\\in\\mathcal\{C\}\}\\sum\_\{x\_\{i\}\\in\\mathcal\{N\}\_\{k\}\(z\)\}\\mathbbm\{1\}\(y\_\{i\}=c\)where𝒟\\mathcal\{D\}is the set of all evaluation points and𝒞\\mathcal\{C\}the set of all classes\. The implementation runs on GPU and is heavily based on the released code of CAPI\(Darcetet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib8)\)\.

In practice, these evaluations are computationally negligible\. For a 250M parameter model, the entire linear and kNN probing process on GLUE takes less than 5 minutes on a single H100\. The cost is dominated by the forward pass over the dataset\.

### 4\.2MTEB frozen evaluation

We evaluate the richness and adaptability of our frozen representation using the Massive Text Embedding Benchmark\(Muennighoffet al\.,[2023](https://arxiv.org/html/2607.04011#bib.bib6)\), which spans seven distinct task downstream categories: Classification, Clustering, Pair Classification, Semantic Textual Similarity \(STS\), Reranking, Retrieval, and Summarization\.

#### Contrastive probing\.

The above downstream tasks require sentence/document representations and a coherent, well\-structured representation space, which is not tackled by the pre\-training objective\. Contrastive finetuning aims to address these gaps\. While existing literature typically performs this via contrastive finetuning on an unfrozen backbone, we explore two configurations: a standard unfrozen backbone and a frozen\-backbone approach where only an attention based pooler is optimized on top of fixed features\.

The token\-level encoded representations are pooled into a single sentence/document representation via a learnable lightweight adapter, consisting of a few Transformer blocks with cross\-attention\. Let the pooled representations of a passage \(resp\. query\) bepp\(resp\.qq\)\. Given a queryqq, the contrastive loss, aims to pull closer a related passageppand the queryqqin representation space while pushing away semantically similar unrelated passagesp−p^\{\-\}\(hard\-negatives\)\. The contrastive loss, as defined inChenet al\.\([2020](https://arxiv.org/html/2607.04011#bib.bib9)\), is

ℒc=−1N​∑i=1Nlog⁡eϕ​\(qi,pi\)eϕ​\(qi,pi\)\+∑n∈𝒮ieϕ​\(qi,pi​n−\)\\mathcal\{L\}\_\{c\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\log\\frac\{e^\{\\phi\(q\_\{i\},p\_\{i\}\)\}\}\{e^\{\\phi\(q\_\{i\},p\_\{i\}\)\}\+\\sum\_\{n\\in\\mathcal\{S\}\_\{i\}\}e^\{\\phi\(q\_\{i\},p\_\{in\}^\{\-\}\)\}\}\(5\)whereϕ​\(qi,pi\)\\phi\(q\_\{i\},p\_\{i\}\)is the cosine similarity between a queryqiq\_\{i\}and a passagepip\_\{i\}, andpi​n−p\_\{in\}^\{\-\}denotes the hard negatives for each query\.

Since CrossBERT already features an adaptable component \(the predictor\), we re\-purpose it to warm\-start the adapter for the contrastive learning phase, while keeping the encoder frozen\. As detailed in our ablation study \([AppendixD](https://arxiv.org/html/2607.04011#A4)\), this strategy accelerates convergence and improves MTEB scores by≈2\\approx 2%\.

### 4\.3Evaluations at different scales

Model performance and optimal hyperparameters follow predictable power\-law trends relative to the total compute budgetCC\(Kaplanet al\.,[2020](https://arxiv.org/html/2607.04011#bib.bib30); Biet al\.,[2024](https://arxiv.org/html/2607.04011#bib.bib32)\)\. We defineCCas:

whereDDis the number of pre\-training tokens andFNF\_\{N\}are the FLOPs per token for a forward\-backward pass\. For a standard BERT transformer with sequence lengthSS, layersLL, and non\-embedding parametersNN\(Dervishiet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib1)\)we have:

FN=6​N\+12​d​L​SF\_\{N\}=6N\+12dLS\(7\)However, CrossBERT modifiesFNF\_\{N\}based on the masking ratiorr, linearly interpolating between the encoder \(FNencF^\{\\text\{enc\}\}\_\{N\}\) and the cross\-attention predictor \(FNpredF^\{\\text\{pred\}\}\_\{N\}\):

FN=FNenc​\(1−r\)\+FNpred​rF\_\{N\}=F^\{\\text\{enc\}\}\_\{N\}\(1\-r\)\+F^\{\\text\{pred\}\}\_\{N\}r\(8\)This demonstrates that as the masking ratio increases, the total computational cost linearly interpolates between the encoder and the predictor\. To scale up we need to increase model sizesFNF\_\{N\}and dataset sizesDDin tandem\.

## 5Experiments

### 5\.1Setup

#### Models\.

We train three models that share the same corpus, tokenizer, and encoder backbone: BERT, Electra, and CrossBERT\. BERT and Electra are flat, whereas CrossBERT is bipartite \([Section3](https://arxiv.org/html/2607.04011#S3)\)\. BERT and CrossBERT use the MLM objective of[Section2](https://arxiv.org/html/2607.04011#S2), while Electra replaces it with replaced\-token detection \(RTD\)\(Clarket al\.,[2020](https://arxiv.org/html/2607.04011#bib.bib39)\): a small auxiliary generator substitutes a fraction of the input tokens with plausible alternatives, and the encoder predicts at each position whether the token is original or replaced\. For context, we also report the published results of ModernBERT\(Warneret al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib2)\), NeoBERT\(Bretonet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib3)\), DeBERTaV3\(Heet al\.,[2023](https://arxiv.org/html/2607.04011#bib.bib40)\), and OptiBERT\(Dervishiet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib1)\), all of which are trained with substantially more data and compute\.

#### Data\.

We choose a subset of DCLM\(Liet al\.,[2024](https://arxiv.org/html/2607.04011#bib.bib17)\)as our training corpus \(≈4​T\\approx 4\\text\{T\}tokens\)\. Across all experiments, the masking ratio is set to20%20\\%for BERT and50%50\\%for CrossBERT\. Additionally, all CrossBERT models leverage CMS \(see[Section3\.2](https://arxiv.org/html/2607.04011#S3.SS2)\)\. All models rely on the RoBERTa tokenizer\(Liuet al\.,[2019](https://arxiv.org/html/2607.04011#bib.bib26)\)\.

#### Codebase\.

Our codebase is based on PyTorch\(Paszkeet al\.,[2019](https://arxiv.org/html/2607.04011#bib.bib27)\)and Lingua\(Videauet al\.,[2024](https://arxiv.org/html/2607.04011#bib.bib28)\), opting for Fully Sharded Data Parallelism \(FSDP\) andtorch\.compilefor maximum throughput and reduced memory footprint\.Handling Dynamic Shapes:Compilation requires static graphs, which conflicts with random MLM masking\. We resolve this by enforcing a fixed count of masked tokens per GPU \(Per GPU Batch size×\\timesMask Ratio\) and permuting each mask in the sequence via random permutations \(torch\.randperm\)\. This ensures static tensor shapes without compromising masking randomness\.

![Refer to caption](https://arxiv.org/html/2607.04011v1/x4.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x5.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x6.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x7.png)

![Refer to caption](https://arxiv.org/html/2607.04011v1/x8.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x9.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x10.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x11.png)

Figure 3:Scaling trend of BERT vs CrossBERT after contrastive finetuning under frozen and unfrozen backbone for MTEB\(eng, v2\)\. All models are trained for one epoch on MS\-MARCO as described in[Section5\.1](https://arxiv.org/html/2607.04011#S5.SS1)
#### Hyperparameters\.

FollowingBiet al\.\([2024](https://arxiv.org/html/2607.04011#bib.bib32)\), we fit power laws for both Batch Size \(BSZ\) in total tokens and Learning Rate \(LR\) sweeping model sizes ranging from \(50M to 700M\) with a LR \(resp\. BSZ\) logspaced from10−410^\{\-4\}to10−210^\{\-2\}\(resp\.10410^\{4\}to5×1065\\times 10^\{6\}\)\. After keeping only the top 3 performing models for the fitting, we obtain the following results for CrossBERT:

BSZCrossBERT​\(C\)\\displaystyle\\text\{BSZ\}\_\{\\text\{CrossBERT\{\}\}\}\(C\)=104×C0\.092\\displaystyle=0^\{4\}\\times C^\{0\.092\}LRCrossBERT​\(C\)\\displaystyle\\text\{LR\}\_\{\\text\{CrossBERT\{\}\}\}\(C\)=564\.6×C−0\.279\\displaystyle=646\\times C^\{\-0\.279\}For the BERT setup, we simply reuse the hyperparameters found by\(Dervishiet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib1)\)

BSZBERT​\(C\)\\displaystyle\\text\{BSZ\}\_\{\\text\{BERT\}\}\(C\)=17\.38×C0\.24\\displaystyle=17\.38\\times C^\{0\.24\}LRBERT​\(C\)\\displaystyle\\text\{LR\}\_\{\\text\{BERT\}\}\(C\)=69\.18×C−0\.24\\displaystyle=69\.18\\times C^\{\-0\.24\}[AppendixH](https://arxiv.org/html/2607.04011#A8)shows the hyperparameter heatmap for the CrossBERT sweeping\.

#### Eval on GLUE\.

For Linear Probing, we fit linear heads on frozen features along logarithmic data regimes \(\{1,…,104\}\\\{1,\\dots,10^\{4\}\\\}samples\) and L2\-regularization strengths \(λ∈\[1,104\]\\lambda\\in\[1,10^\{4\}\]\)\. For each task we report the best score across allλ\\lambdafor the largest available sample size\. For kNN Probing, non\-parametric evaluation with kNN are conducted using diverse configurations by sweeping across distance metrics \(L2L\_\{2\}, Cosine\) and neighborhood sizes \(k∈\{1,3,10,30\}k\\in\\\{1,3,10,30\\\}\)\. We report the maximum score across allkkand distances\.

#### Eval on MTEB\.

We use the MS\-MARCO training set \(500k queries\) with hard negatives\. No instruction templates are applied\. We finetune contrastively for one epoch, with a batch size of 512 and a learning rate of5×10−55\\times 10^\{\-5\}\(cosine decay to5×10−75\\times 10^\{\-7\}\)\. To be comparable with current published models, we use both an unfrozen and frozen backbone\.

#### Single\-scale runs\.

All three models share the same encoder backbone \(28 layers, hidden dimension 768\); CrossBERT additionally couples this encoder to a 6\-layer cross\-attention predictor\. The models are trained on 50B tokens with a data\-to\-model ratio of35:135:1that deliberately exceeds the compute\-optimal≈15:1\\approx 15:1to avoid undertraining\. The learning rate is fixed at6×10−46\\times 10^\{\-4\}and the global batch size at≈393​k\\approx 393\\text\{k\}\.

#### Scaling sweep\.

Dervishiet al\.[2025](https://arxiv.org/html/2607.04011#bib.bib1)showed that the data\-to\-model ratioFN/DF\_\{N\}/Dheavily impacts performance and should be kept around 10:1 to 100:1\. In all our scaling experiments we fix the ratio at20:120:1and generate a suite of models spanning a total compute range from2×10182\\times 10^\{18\}to1×10211\\times 10^\{21\}FLOPs\. To optimize performance at every scale, we determine batch size and learning rate by sweeping values across smaller models \(50​M50\\text\{M\}to700​M700\\text\{M\}\) and extrapolating the optimal settings via a power law fitting\)\. In total we train 8 BERT and 9 CrossBERT models, see[AppendixJ](https://arxiv.org/html/2607.04011#A10)\.

### 5\.2Results: Single\-scale Runs

[Table3](https://arxiv.org/html/2607.04011#S3.T3)reports the single\-scale runs of our three models alongside the current literature\.

#### Training efficiency and throughput\.

CrossBERT demonstrates superior computational efficiency\. In terms of training throughput, it achieves 207k tokens/sec compared to the baseline’s 123k tokens/sec, representing a1\.68×1\.68\\timesspeedup in wall\-clock time on H100 GPUs\. Furthermore, when compared to the existing literature, CrossBERT remains highly competitive despite utilizing approximately100×100\\timesless total compute than models like ModernBERT and NeoBERT\. Against OptiBERT, a model of similar scale, CrossBERT requires≈40%\\approx 40\\%fewer FLOPs while delivering higher performance on both GLUE and MTEB\(eng, v1\)\.

#### Full\-finetuning performances\.

Despite this massive reduction in compute, CrossBERT does not compromise on quality\. On the GLUE benchmark, it matches our robust baseline \(86\.686\.6vs\.86\.486\.4\)\. More notably, on the MTEB\(eng, v1\) benchmark \(Full Contrastive Finetuning\), CrossBERT achieves the highest average score of54\.5, outperforming both NeoBERT \(51\.351\.3\) and ModernBERT \(46\.946\.9\) which used much more finetuning data\.

![Refer to caption](https://arxiv.org/html/2607.04011v1/x12.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x13.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x14.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x15.png)

![Refer to caption](https://arxiv.org/html/2607.04011v1/x16.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x17.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x18.png)![Refer to caption](https://arxiv.org/html/2607.04011v1/x19.png)

Figure 4:Scaling trends of BERT vs CrossBERT using linear and kNN probing for GLUE under frozen backbone\(see[Section5\.1](https://arxiv.org/html/2607.04011#S5.SS1)\)\.
#### Robustness of frozen representations\.

The most significant advantage of CrossBERT lies in the versatility of its frozen features\. We observe three critical behaviors:

- •Minimal degradation:When switching from full finetuning to frozen adaptation, the standard BERT baseline suffers a substantial drop of over 10 points on MTEB\(eng, v1\) \(53\.9→42\.153\.9\\to 42\.1\)\. In contrast, CrossBERT retains 99% of its performance, scoring 53\.6 in the frozen setting\. This indicates that the pretrained features are easily adaptable\.
- •Retrieval capability:This robustness is most visible in the Retrieval task, where CrossBERT more than doubles the score of the baseline \(40\.740\.7vs\.19\.219\.2\), proving that the model captures semantic similarity without needing deep task\-specific adaptation\.
- •Linear separability:On the GLUE Linear Probe, CrossBERT surpasses the BERT baseline by a large margin \(\+6\.4\+6\.4points\)\. This confirms that our approach prevents the over\-specialization often seen in standard MLMs, producing high\-level abstractions that are linearly separable and directly usable for downstream tasks\. While a performance gap of≈13\\approx 13points remains between the linear probe and full finetuning, this is partly attributable to the simplicity of averaging encoder outputs; utilizing more sophisticated probing mechanisms, such as attention\-based pooling, would likely reduce this gap further\. Electra scores even higher than CrossBERT on this probe, but this advantage does not carry over to the sentence\-embedding tasks examined next\.

#### MLM versus RTD on a flat backbone\.

On token\-level probes, Electra outperforms BERT and CrossBERT, reaching87\.887\.8on GLUE full finetuning and76\.976\.9on the GLUE linear probe\. However, on sentence\-level embedding tasks it collapses\. The unfrozen backbone benchmarks at49\.049\.0on MTEB\(eng,v1\) whereas the frozen one drops to22\.422\.4with a Retrieval score of0\.00\.0\. This collapse mirrors prior findings that RTD distorts sentence\-embedding geometry\(Repet al\.,[2024](https://arxiv.org/html/2607.04011#bib.bib41); Warneret al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib2)\)\. Two conclusions follow\. First, neither swapping the objective on a flat backbone \(Electra\) nor keeping MLM on a flat backbone \(BERT\) yields versatile frozen representations; the bipartite separation of representation from reconstruction is what produces them\. Second, GLUE\-style classification probes alone cannot diagnose representation quality, since MTEB exposes failures that GLUE hides\.

### 5\.3Results: Scaling Sweep

[Figure1](https://arxiv.org/html/2607.04011#S0.F1)aggregates the GLUE and MTEB\(eng,v2\) scores of BERT and CrossBERT, while[Figures4](https://arxiv.org/html/2607.04011#S5.F4)and[3](https://arxiv.org/html/2607.04011#S5.F3)break them down per task\. Our sweep reveals four insights into the scaling behavior of masked language models\.

#### Degradation of frozen BERT representations\.

The most striking trend is the sharp divergence in scaling laws between the two models\. While standard BERT benefits from scaling when it is unfrozen and fully finetuned, itsfrozenperformance suffers a brutal drop as model size increases\. Specifically, for the two largest configurations \(\>500\>500M parameters\), BERT’s frozen performance drops below that of the smallest model in the sweep \(10×10\\timessmaller\)\. Since the data\-to\-model ratio \(20:1\) is held above the compute\-optimal ratio \(15:1\), undertraining cannot explain the drop\. This confirms a severe misalignment between the standard MLM objective and semantic embedding tasks: as the model scales, it becomes increasingly specialized for token reconstruction at the expense of versatile, high\-level abstractions\.

#### Task\-specific sensitivity on GLUE\.

This degradation manifests non\-uniformly across tasks \([Figure4](https://arxiv.org/html/2607.04011#S5.F4)\)\. On the GLUE benchmark, the sudden decline of performance does not necessarily appear at the same point\. For example MNLI accuracy drops only for the last model while performances for the rest seem to drop earlier\.

#### MTEB\(eng, v2\) superior scaling and saturation profiles\.

In contrast, CrossBERT exhibits robust, monotonic scaling across all metrics\. Most notably, CrossBERT fundamentally alters the relationship between pre\-training and adaptation:

- •The adaptation gap & equalization on MTEB\(eng, v2\):Standard BERT requires full finetuning to bridge a massive performance deficit \(≈15\\approx 15points\) between its frozen and unfrozen states\. While this heavy downstream adaptation can eventually equalize performance, i\.e\. masking pre\-training deficiencies by bringing BERT closer to CrossBERT, it comes at a significant compute cost\. In contrast, CrossBERT’s frozen representations are naturally aligned, sitting at worse around one point below the fully fine\-tuned optimum\.
- •Structural retrieval capability:We observe a fundamental distinction in retrieval tasks\. Standard MLM effectively flatlines near zero at all scales, indicating a structural inability to learn dense retrieval without supervision\. In contrast, CrossBERT builds these capabilities naturally, scaling linearly with compute\.
- •Frozen outperforming unfrozen finetuning:At larger scales, CrossBERT with frozen adaptation begins to outperform even the unfrozen baselines\. This indicates that CrossBERT scales more effectively than standard BERT, avoiding early saturation\. It produces representations that are naturally richer, rendering the heavy process of unfrozen finetuning unnecessary, and eventually inferior, to a lightweight adaptation of the frozen features\.

#### Training stability\.

Self\-supervised pre\-training can suffer from unstable optimization, so we monitor training across the full scaling range\. Two observations indicate that CrossBERT trains stably\. First, the pre\-training validation loss decreases smoothly and tracks the expected scaling law at every compute budget \([AppendixF](https://arxiv.org/html/2607.04011#A6)\)\. Second, downstream probing performance rises steadily over the course of training rather than oscillating or collapsing \([Figure6](https://arxiv.org/html/2607.04011#A7.F6)\)\. We observe no divergence across scales\. In contrast higher scale BERT trainings suffer from significant instability and loss spikes \(see[AppendixI](https://arxiv.org/html/2607.04011#A9)\)\.

## 6Related Works

#### Standard Encoders\.

Since BERT\(Devlinet al\.,[2019](https://arxiv.org/html/2607.04011#bib.bib5)\)and RoBERTa\(Liuet al\.,[2019](https://arxiv.org/html/2607.04011#bib.bib26)\), encoders have relied onflatarchitectures where representation and reconstruction are mixed\. Recent updates like ModernBERT\(Warneret al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib2)\)and NeoBERT\(Bretonet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib3)\)scale this design but inherit its structural inefficiency\. As a result, brute\-force scaling of this approach yields diminishing returns for representation quality\(Dervishiet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib1)\)\.

#### Contrastive Learning\.

To compensate for this misalignment, the field relies on heavy post\-training\(Gaoet al\.,[2021](https://arxiv.org/html/2607.04011#bib.bib36); Wanget al\.,[2022](https://arxiv.org/html/2607.04011#bib.bib18); Liet al\.,[2023](https://arxiv.org/html/2607.04011#bib.bib37)\)or token\-level objectives like MEXMA\(Janeiroet al\.,[2025](https://arxiv.org/html/2607.04011#bib.bib34)\)\. These methods effectively treat the symptoms, but we posit that a better aligned pre\-training can reduce the amount of adaptation needed to obtain the final model\.

#### Architecture Design\.

In computer vision, Masked Autoencoders\(Heet al\.,[2022](https://arxiv.org/html/2607.04011#bib.bib25)\)established the efficacy of asymmetric designs, where a lightweight decoder reconstructs pixels from highly masked inputs\.Fuet al\.\([2024](https://arxiv.org/html/2607.04011#bib.bib7)\)further demonstrated that reducing interaction between representation and reconstruction, using cross attention only, enhances produced features\. In NLP, while T5\(Raffelet al\.,[2020](https://arxiv.org/html/2607.04011#bib.bib35)\)also utilizes a bipartite structure, it relies on a heavy decoder optimized for autoregressive text generation employing both self and cross attention\. In contrast, CrossBERT adopts asymmetry strictly for representation learning: rather than generating text, we employ a lightweight, cross attention only, predictor solely to offload the reconstruction burden, ensuring the encoder optimizes for semantic abstraction rather than token prediction\.

## 7Limitations

Our analysis is empirical\. We identify the failure mode of flat MLM through controlled experiments \([Table1](https://arxiv.org/html/2607.04011#S3.T1)and[Figure1](https://arxiv.org/html/2607.04011#S0.F1)\), but we do not give a theoretical account of why coupling representation and reconstruction degrades frozen features as models grow\. Our conclusions also rest on models no larger than a few billion parameters; every trend we observe is monotonic across this range, yet we cannot formally exclude qualitatively different behavior at the much larger scales typical of decoders\. Finally, we find that RTD \(Electra\) yields strong token\-level probes but poor sentence embeddings, and a mechanistic explanation of this collapse lies outside the scope of this work\.

## 8Conclusion & Future Work

We revisited the design of text encoders by shifting the evaluation focus from full finetuning to frozen representation quality\. This change in perspective revealed that flat BERT\-like architectures trained with MLM suffer from a fundamental misalignment: as compute scales, representations become increasingly unexploitable for downstream tasks, overspecializing on reconstruction at the expense of versatility\. CrossBERT resolves this by insulating representation learning from the token prediction task\. Beyond improving training efficiency, our results show an interesting trend: at large scales, CrossBERT produces frozen features that outperform fully finetuned baselines\. This result challenges the expensive downstream contrastive adaptation of standard encoders, demonstrating that the right pre\-training incentives can produce significantly richer representations needing less heavy adaptation to be effective\.

Several directions follow naturally from these findings\. The bipartite design is agnostic to the reconstruction objective, so MLM could be swapped for alternative pretext tasks better suited to representation learning\. A systematic comparison with T5\-style pre\-training, where the predictor is replaced by an auto\-regressive decoder would be an interesting avenue to explore\.

## Acknowledgements

We thank Badr Youbi Idrissi and João Maria Janeiro for the helpful discussions and feedback that shaped this work\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- M\. Assran, Q\. Duval, I\. Misra, P\. Bojanowski, P\. Vincent, M\. Rabbat, Y\. LeCun, and N\. Ballas \(2023\)Self\-supervised learning from images with a joint\-embedding predictive architecture\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 15619–15629\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1)\.
- A\. Bardes, J\. Ponce, and Y\. LeCun \(2021\)Vicreg: variance\-invariance\-covariance regularization for self\-supervised learning\.arXiv preprint arXiv:2105\.04906\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1)\.
- X\. Bi, D\. Chen, G\. Chen, S\. Chen, D\. Dai, C\. Deng, H\. Ding, K\. Dong, Q\. Du, Z\. Fu,et al\.\(2024\)Deepseek llm: scaling open\-source language models with longtermism\.arXiv:2401\.02954\.Cited by:[§4\.3](https://arxiv.org/html/2607.04011#S4.SS3.p1.2),[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px4.p1.4)\.
- D\. Bolya, P\. Huang, P\. Sun, J\. H\. Cho, A\. Madotto, C\. Wei, T\. Ma, J\. Zhi, J\. Rajasegaran, H\. Rasheed,et al\.\(2025\)Perception encoder: the best visual embeddings are not at the output of the network\.arXiv preprint arXiv:2504\.13181\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1)\.
- L\. L\. Breton, Q\. Fournier, M\. E\. Mezouar, J\. X\. Morris, and S\. Chandar \(2025\)NeoBERT: a next\-generation bert\.arXiv preprint arXiv:2502\.19587\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.04011#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px1.p1.1)\.
- M\. Caron, H\. Touvron, I\. Misra, H\. Jégou, J\. Mairal, P\. Bojanowski, and A\. Joulin \(2021\)Emerging properties in self\-supervised vision transformers\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 9650–9660\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1)\.
- T\. Chen, S\. Kornblith, M\. Norouzi, and G\. Hinton \(2020\)A simple framework for contrastive learning of visual representations\.InInternational conference on machine learning,pp\. 1597–1607\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1),[§4\.2](https://arxiv.org/html/2607.04011#S4.SS2.SSS0.Px1.p2.6)\.
- K\. Clark, M\. Luong, Q\. V\. Le, and C\. D\. Manning \(2020\)ELECTRA: pre\-training text encoders as discriminators rather than generators\.External Links:2003\.10555,[Link](https://arxiv.org/abs/2003.10555)Cited by:[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px1.p1.1)\.
- T\. Darcet, F\. Baldassarre, M\. Oquab, J\. Mairal, and P\. Bojanowski \(2025\)Cluster and predict latent patches for improved masked image modeling\.arXiv preprint arXiv:2502\.08769\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1),[§4\.1](https://arxiv.org/html/2607.04011#S4.SS1.SSS0.Px2.p1.7)\.
- M\. Dervishi, A\. Allauzen, G\. Synnaeve, and Y\. LeCun \(2025\)Training compute\-optimal transformer encoder models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 35602–35617\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1804/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1804),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p1.1),[§4\.3](https://arxiv.org/html/2607.04011#S4.SS3.p1.7),[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px4.p1.5),[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px8.p1.6),[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§2](https://arxiv.org/html/2607.04011#S2.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px1.p1.1)\.
- L\. Fu, L\. Lian, R\. Wang, B\. Shi, X\. Wang, A\. Yala, T\. Darrell, A\. A\. Efros, and K\. Goldberg \(2024\)Rethinking patch dependence for masked autoencoders\.arXiv preprint arXiv:2401\.14391\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1),[§1](https://arxiv.org/html/2607.04011#S1.p3.1),[§2](https://arxiv.org/html/2607.04011#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px3.p1.1)\.
- T\. Gao, X\. Yao, and D\. Chen \(2021\)Simcse: simple contrastive learning of sentence embeddings\.arXiv preprint arXiv:2104\.08821\.Cited by:[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px2.p1.1)\.
- Q\. Garrido, M\. Assran, N\. Ballas, A\. Bardes, L\. Najman, and Y\. LeCun \(2024\)Learning and leveraging world models in visual representation learning\.arXiv preprint arXiv:2403\.00504\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1)\.
- K\. He, X\. Chen, S\. Xie, Y\. Li, P\. Dollár, and R\. Girshick \(2022\)Masked autoencoders are scalable vision learners\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 16000–16009\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1),[§1](https://arxiv.org/html/2607.04011#S1.p3.1),[§2](https://arxiv.org/html/2607.04011#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2607.04011#S3.SS1.p2.1),[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px3.p1.1)\.
- P\. He, J\. Gao, and W\. Chen \(2023\)DeBERTaV3: improving deberta using electra\-style pre\-training with gradient\-disentangled embedding sharing\.External Links:2111\.09543,[Link](https://arxiv.org/abs/2111.09543)Cited by:[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px1.p1.1)\.
- J\. M\. Janeiro, B\. Piwowarski, P\. Gallinari, and L\. Barrault \(2025\)MEXMA: token\-level objectives improve sentence representations\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 23960–23995\.Cited by:[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px2.p1.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv:2001\.08361\.Cited by:[§4\.3](https://arxiv.org/html/2607.04011#S4.SS3.p1.2)\.
- J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Y\. Gadre, H\. Bansal, E\. Guha, S\. S\. Keh, K\. Arora,et al\.\(2024\)Datacomp\-lm: in search of the next generation of training sets for language models\.Advances in Neural Information Processing Systems37,pp\. 14200–14282\.Cited by:[Figure 5](https://arxiv.org/html/2607.04011#A6.F5),[Figure 5](https://arxiv.org/html/2607.04011#A6.F5.3.2),[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px2.p1.3)\.
- Z\. Li, X\. Zhang, Y\. Zhang, D\. Long, P\. Xie, and M\. Zhang \(2023\)Towards general text embeddings with multi\-stage contrastive learning\.arXiv preprint arXiv:2308\.03281\.Cited by:[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px2.p1.1)\.
- Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov \(2019\)Roberta: a robustly optimized bert pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px2.p1.3),[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px1.p1.1)\.
- N\. Muennighoff, N\. Tazi, L\. Magne, and N\. Reimers \(2023\)Mteb: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 2014–2037\.Cited by:[§4\.2](https://arxiv.org/html/2607.04011#S4.SS2.p1.1),[§4](https://arxiv.org/html/2607.04011#S4.p3.1)\.
- M\. Oquab, T\. Darcet, T\. Moutakanni, H\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby,et al\.\(2023\)Dinov2: learning robust visual features without supervision\.arXiv preprint arXiv:2304\.07193\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1)\.
- A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga,et al\.\(2019\)Pytorch: an imperative style, high\-performance deep learning library\.Advances in neural information processing systems32\.Cited by:[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px3.p1.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px3.p1.1)\.
- I\. Rep, D\. Dukić, and J\. Šnajder \(2024\)Are ELECTRA’s sentence embeddings beyond repair? the case of semantic textual similarity\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 9159–9169\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.535/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.535)Cited by:[§5\.2](https://arxiv.org/html/2607.04011#S5.SS2.SSS0.Px4.p1.5)\.
- O\. Siméoni, H\. V\. Vo, M\. Seitzer, F\. Baldassarre, M\. Oquab, C\. Jose, V\. Khalidov, M\. Szafraniec, S\. Yi, M\. Ramamonjisoa,et al\.\(2025\)Dinov3\.arXiv preprint arXiv:2508\.10104\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1)\.
- Q\. Sun, Y\. Fang, L\. Wu, X\. Wang, and Y\. Cao \(2023\)Eva\-clip: improved training techniques for clip at scale\.arXiv preprint arXiv:2303\.15389\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p2.1)\.
- M\. Videau, B\. Y\. Idrissi, D\. Haziza, L\. Wehrstedt, J\. Copet, O\. Teytaud, and D\. Lopez\-Paz \(2024\)Meta Lingua: a minimal PyTorch LLM training library\.External Links:[Link](https://github.com/facebookresearch/lingua)Cited by:[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px3.p1.1)\.
- A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. Bowman \(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp\. 353–355\.Cited by:[§4](https://arxiv.org/html/2607.04011#S4.p3.1)\.
- L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. Wei \(2022\)Text embeddings by weakly\-supervised contrastive pre\-training\.arXiv preprint arXiv:2212\.03533\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p1.1),[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px2.p1.1)\.
- B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen,et al\.\(2025\)Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2526–2547\.Cited by:[§1](https://arxiv.org/html/2607.04011#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.04011#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.04011#S5.SS1.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2607.04011#S5.SS2.SSS0.Px4.p1.5),[§6](https://arxiv.org/html/2607.04011#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Wettig, T\. Gao, Z\. Zhong, and D\. Chen \(2023\)Should you mask 15% in masked language modeling?\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 2985–3000\.Cited by:[§3\.2](https://arxiv.org/html/2607.04011#S3.SS2.SSS0.Px2.p1.4)\.

## Appendix APredictor shape design ablation

[Table4](https://arxiv.org/html/2607.04011#A1.T4)analyzes the trade\-off between predictor size and downstream performance\. At the lower bound, the 8M parameter predictor acts as a representational bottleneck, noticeably impairing model quality\. However, increasing capacity to just 25M yields a substantial boost\. Beyond this point, we observe diminishing returns; scaling further to 50M or 85M incurs higher computational costs for only marginal performance gains\. While we ultimately selected the 42M configuration \(one\-fourth of the encoder depth, grey row\) to maintain dimensional alignment with the backbone, the data suggests that the 25M variant remains a highly competitive alternative, promising faster pretraining speed\.

PredictorGLUEParamsDimLayersMNLIQNLIQQPRTESST2MRPCCoLASTSAvg\.8M1921258\.278\.480\.164\.687\.672\.845\.382\.271\.225M3841261\.581\.681\.462\.088\.578\.049\.585\.173\.450M5761262\.881\.781\.562\.189\.278\.947\.886\.673\.885M7681263\.481\.981\.963\.991\.076\.048\.486\.774\.242M768661\.681\.881\.464\.390\.677\.947\.884\.873\.8Table 4:Impact of varying the shape and size of the predictor\. The encoder has 236\.8M parameters \(768 dim, 28 layers\)\. All model are trained on same exact setup as[Table3](https://arxiv.org/html/2607.04011#S3.T3)
## Appendix BComplementary Masking ablation

ModelCMSMNLIQNLIQQPRTESST2MRPCCoLASTSBAvg\.CrossBERT 279M✗61\.481\.481\.362\.888\.974\.248\.785\.873\.1CrossBERT 279M✓61\.681\.881\.464\.290\.677\.947\.884\.873\.8Table 5:Effect of CMS on GLUE Linear probe\. All results are obtained under the same settings of[Table3](https://arxiv.org/html/2607.04011#S3.T3)
## Appendix CBERT Masking ablation

BERT maskCoLAMNLIMRPCQNLIQQPRTESST2STSBAvg\.20%27\.458\.676\.774\.877\.759\.984\.979\.567\.430%18\.857\.073\.074\.177\.954\.280\.777\.164\.140%11\.053\.669\.173\.575\.057\.075\.173\.461\.065%22\.650\.067\.270\.073\.455\.275\.165\.759\.9Table 6:GLUE scores for BERT models trained with different masking rates\.
## Appendix DPredictor MTEBv2 contrastive finetuning ablation

ModelWarm\-startClass\.Clust\.PairClass\.Rerank\.Retriev\.STSSumm\.Avg\.OverallCrossBERT 279M✗68\.738\.376\.443\.437\.868\.928\.751\.853\.7CrossBERT 279M✓70\.540\.877\.944\.143\.971\.726\.753\.756\.7Table 7:Contrastive finetuning under frozen backbones on MSMarco for one epochs as described in[Section5\.1](https://arxiv.org/html/2607.04011#S5.SS1)
## Appendix EMTEB\(eng,v2\) Results of the Single\-scale runs\.

MTEB\(eng, v2\) Full Contrastive Finetuning on MS\-MARCOClass\.Clust\.PairClass\.Rerank\.Retriev\.STSSumm\.Avg\.OverallBERT239M6\.8e19123k67\.739\.380\.843\.945\.775\.328\.754\.557\.3CrossBERT279M4\.1e19207k72\.235\.580\.544\.844\.274\.632\.054\.857\.0Electra258M7\.5e19116k62\.127\.777\.948\.834\.472\.429\.150\.349\.0MTEB\(eng, v2\) Contrastive Finetuning over frozen features on MS\-MARCOBERT239M6\.8e19123k64\.937\.364\.539\.420\.561\.631\.845\.745\.9CrossBERT279M4\.1e19207k70\.440\.877\.944\.143\.971\.726\.753\.656\.7Electra258M7\.5e19116k43\.929\.535\.433\.00\.331\.919\.427\.626\.1Table 8:MTEB\(eng,v2\) results of our models under contrastive finetuning over frozen and unfrozen backbone \(full\-finetuning\)\.
## Appendix FLearning curve of scaling laws

![Refer to caption](https://arxiv.org/html/2607.04011v1/x20.png)

![Refer to caption](https://arxiv.org/html/2607.04011v1/x21.png)

Figure 5:Learning curves for different BERT\(left\) and CrossBERT\(right\) setup specified in model list[AppendixJ](https://arxiv.org/html/2607.04011#A10)\. Each dot represents the validation loss on wikipedia and dclm\(Liet al\.,[2024](https://arxiv.org/html/2607.04011#bib.bib17)\)\.
## Appendix GDownstream performance evolution during training

![Refer to caption](https://arxiv.org/html/2607.04011v1/x22.png)Figure 6:Average classification performance across 10 tasks monitored during training for BERT \(red,■\\blacksquare\) and CrossBERT \(blue,∙\\Large\\bullet\)\.
## Appendix HLearning rate and batch size sweeps

![Refer to caption](https://arxiv.org/html/2607.04011v1/x23.png)

![Refer to caption](https://arxiv.org/html/2607.04011v1/x24.png)

![Refer to caption](https://arxiv.org/html/2607.04011v1/x25.png)

Figure 7:CrossBert heatmap for lr and batch size, Left: flops vs lr, Middle: flops vs batch size, Right: lr vs batch size, the size of the point is proportionnal to total compute budget spend for training\.
## Appendix ILarge scale training stability

Because scaling BERT resulted in a severe degradation of downstream performance, we further investigated this collapse across various training setups\. To determine if this instability is specific to BERT, we additionally scaled Electra to 1B parameters\. As shown in[Table9](https://arxiv.org/html/2607.04011#A9.T9), both BERT and Electra exhibit a similar performance collapse at the 1B scale\. For BERT, altering standard hyperparameters—including data and model seeds, learning rate, and batch size—failed to prevent the degradation\. While specific hyperparameter tuning allowed Electra to avoid collapsing, this extreme sensitivity highlights the inherent brittleness of training standard ’flat’ architectures at scale\.

Conversely, CrossBERT is highly robust, achieving strong out\-of\-the\-box performance without requiring such exhaustive hyperparameter sweeps\. When scaling further to 2B parameters, standard configurations initially failed to converge\. However, we found that simply reducing the initialization standard deviation from the conventional 0\.02 \(standard across ViT and BERT implementations\) to 0\.015 successfully stabilized the training\. Furthermore, while Electra historically demonstrates superior performance at smaller scales, CrossBERT 1B significantly outperforms Electra 1B on GLUE, confirming the superior scaling trajectory of our proposed architecture\.

ModelMNLIQNLIQQPRTESST2MRPCCOLASTSBAvg\.BERT 1B60\.378\.673\.560\.371\.270\.819\.370\.163\.0BERT 1B\*59\.681\.376\.663\.979\.772\.312\.371\.664\.6Electra 1B59\.080\.778\.664\.672\.580\.416\.779\.866\.5Electra 1B\*75\.285\.985\.568\.991\.282\.165\.289\.480\.4CrossBERT 1B76\.989\.085\.472\.593\.481\.661\.088\.281\.0BERT 2B\*69\.984\.977\.668\.679\.677\.940\.179\.172\.2\* Training done under different hyperparametersTable 9:Zoom on 1B scale model with 1e21 compute on Frozen Glue
## Appendix JList of Models with hyperparameters

BERT Model ConfigurationsIDParams \(M\)FLOPsDimLayersHeadsHead DimStepsTokens \(B\)BSZGPUsGrad AccLRMin LR𝜷1\\boldsymbol\{\\beta\}\_\{1\}𝜷2\\boldsymbol\{\\beta\}\_\{2\}076\.732\.17e\+1857612964150196\.58438272412\.75e\-031\.0e\-60\.900\.951103\.284\.64e\+18640141064182689\.63527360152\.29e\-031\.0e\-60\.900\.952151\.851\.14e\+197681612642478315\.10609280171\.85e\-031\.0e\-60\.900\.953226\.702\.83e\+198961814642928723\.788120321131\.48e\-031\.0e\-60\.900\.954334\.066\.73e\+1910242216643629136\.681010688371\.21e\-031\.0e\-60\.900\.955503\.811\.65e\+2011522891284677157\.471228800659\.72e\-041\.0e\-60\.900\.956795\.274\.24e\+20140830111285857592\.1315728641637\.75e\-041\.0e\-60\.900\.9571208\.241\.00e\+211664331312872072141\.7019660804036\.30e\-041\.0e\-60\.900\.95

Table 10:BERT model configurations across different scales\. Parameters are in millions \(M\), Tokens in billions \(B\)\. BSZ = total batch size, Grad Acc = gradient accumulation steps\.CrossBERT Model ConfigurationsIDParams\(M\)FLOPsEncDimEncLayersEncHeadsEnc HeadDimPredDimPredLayersPredHeadsStepsTokens\(B\)BSZGPUsGradAccLRMinLR𝜷1\\boldsymbol\{\\beta\}\_\{1\}𝜷2\\boldsymbol\{\\beta\}\_\{2\}061\.801\.92e\+18640151064640310137456\.19450560154\.51e\-031\.0e\-60\.900\.95193\.624\.91e\+18768171264768412197499\.91501760173\.47e\-031\.0e\-60\.900\.952138\.581\.15e\+198961914648964142697815\.195632001112\.73e\-031\.0e\-60\.900\.953205\.572\.72e\+19102423166410245163985023\.345857281132\.15e\-031\.0e\-60\.900\.954315\.616\.83e\+1911522991281152795369136\.95688128371\.67e\-031\.0e\-60\.900\.955494\.241\.70e\+201408311112814087117914158\.35737280651\.29e\-031\.0e\-60\.900\.956757\.474\.07e\+2016643413128166481311466090\.177864321631\.01e\-031\.0e\-60\.900\.9571176\.299\.99e\+201920411512819201015176106141\.388028165619\.00e\-041\.0e\-60\.900\.9585411\.75\.0e212560522012825601320304688319\.49104857612816\.00e\-041\.0e\-60\.900\.95

Table 11:CrossBERT model configurations across different scales\. CrossBERT includes both an encoder and a predictor\. Parameters are in millions \(M\), Tokens in billions \(B\)\. BSZ = total batch size, Grad Acc = gradient accumulation steps\.