Asymmetric Within-Document Predictive Learning for Scientific Document Representation

# Asymmetric Within-Document Predictive Learning for Scientific Document Representation
Source: [https://arxiv.org/html/2608.28625](https://arxiv.org/html/2608.28625)
You Zuo\\up1,2 Éric de la Clergerie\\up2 Benoît Sagot\\up2 \(1\) Questel, Paris, France \(2\) Inria, Paris, France firstname\.lastname@inria\.fr

###### Abstract

Apprentissage prédictif asymétrique intra\-document pour la représentation de documents scientifiques

\\NoAutoSpacing

Abstract

We study predictive pretraining for scientific document representation using the discourse structure of papers\. We propose SciJEPA, a citation\-free framework that learns through asymmetric within\-document prediction: title and abstract representations are used to predict method representations, and method representations are used to predict conclusion representations\. Experiments on RELISH, high\-influence citation, SciDocs, and cite prediction show that plain predictive training is viable but weaker than a controlled contrastive baseline using the same section pairs\. Adding Sliced Isotropic Gaussian Regularization \(SIGReg\) substantially improves performance and narrows this gap\. The effect of regularization is task\-dependent: moderate SIGReg helps fine\-grained ranking, while stronger regularization can weaken local alignment\. We further show that different encoding branches support different retrieval regimes\. These results position within\-document predictive learning as a promising citation\-free complement for scientific document representation, provided that embedding geometry is carefully controlled\.

Nous étudions le pré\-entraînement prédictif pour la représentation de documents scientifiques en exploitant la structure discursive des articles\. Nous proposons SciJEPA, un cadre sans supervision par citations qui apprend par prédiction asymétrique intra\-document : les représentations du titre et du résumé sont utilisées pour prédire les représentations de la section méthodologique, puis les représentations de la section méthodologique pour prédire celles de la conclusion\. Des expériences sur RELISH, high\-influence citation, SciDocs et cite prediction montrent que l’apprentissage prédictif seul est viable, mais reste inférieur à une baseline contrastive contrôlée utilisant les mêmes paires de sections\. L’ajout de la régularisation gaussienne isotrope esquissée \(SIGReg\) améliore nettement les performances et réduit cet écart\. L’effet de la régularisation dépend de la tâche : une SIGReg modérée aide le classement fin, tandis qu’une régularisation plus forte peut affaiblir l’alignement local\. Nous montrons également que différentes branches d’encodage soutiennent différents régimes de recherche\. Ces résultats positionnent l’apprentissage prédictif intra\-document comme un complément prometteur, sans supervision par citations, pour la représentation de documents scientifiques, à condition que la géométrie des plongements soit soigneusement contrôlée\.

Keywords:self\-supervised learning, Joint Embedding Predictive Architecture \(JEPA\), document representation\.

Mots\-clés:apprentissage auto\-supervisé, architecture prédictive à plongements conjoints \(JEPA\), représentation de documents\.

## 1Introduction

Learning effective representations of scientific documents is central to scholarly retrieval, citation recommendation, literature discovery, and document ranking\. Strong scientific document encoders such as SPECTERCohanet al\., \([2020](https://arxiv.org/html/2608.28625#bib.bib9)\)and SPECTER2Singhet al\., \([2023](https://arxiv.org/html/2608.28625#bib.bib21)\)typically encode the title and abstract of a paper and learn from cross\-document supervision derived from citation links or related scholarly tasks\. This strategy is highly effective, especially on citation\-oriented benchmarks, but citation links define a particular supervision signal: they are delayed for newly published papersXinget al\., \([2021](https://arxiv.org/html/2608.28625#bib.bib24)\), vary across fields and article typesRadicchiet al\., \([2008](https://arxiv.org/html/2608.28625#bib.bib20)\), and serve rhetorical functions beyond semantic similarity, such as providing background, acknowledging methods or datasets, supporting claims, or contrasting prior workJurgenset al\., \([2018](https://arxiv.org/html/2608.28625#bib.bib16)\)\. As a result, citation\-supervised encoders combine textual semantics with graph\-induced scholarly relatedness\. In this work, we ask whether useful scientific document representations can be learned from a paper’s content and internal discourse structure alone, without citation links, and whether these citation\-free representations transfer to citation\-related retrieval tasks\. We view this signal as complementary to citation supervision rather than as a replacement when dense citation graphs are available\.

To study this question, we propose SciJEPA, a Joint\-Embedding Predictive Architecture for scientific document representation\. Scientific papers follow a structured progression from problem statement to methodology and findings or implications: the title and abstract describe the problem and contribution, the method section specifies how the problem is addressed, and the conclusion summarizes outcomes and implications\. A useful content\-based representation should support this inference: from a high\-level summary, one should infer the likely methodology, and from the method, plausible conclusions or outcomes\.

SciJEPA operationalizes this idea with two asymmetric within\-document prediction tasks: title and abstract→\\rightarrowmethod, and method→\\rightarrowconclusion\. Following Joint\-Embedding Predictive ArchitecturesLeCunet al\., \([2022](https://arxiv.org/html/2608.28625#bib.bib17)\), SciJEPA predicts the target section in representation space rather than reconstructing it token by token\. This is important because earlier sections do not determine the exact wording or details of later sections, but they do constrain their broader discourse role and semantic content\. The objective therefore encourages the model to learn dependencies between problem statements, methods, and conclusions without citation links or negative examples during pretraining\.

We evaluate SciJEPA on scientific retrieval benchmarks from SciRepEvalSinghet al\., \([2023](https://arxiv.org/html/2608.28625#bib.bib21)\)and compare it with a controlled contrastive baseline using the same corpus, backbone, and section pairs\. Plain predictive training learns useful citation\-free representations but remains weaker than contrastive learning; adding SIGRegBalestriero & LeCun, \([2025](https://arxiv.org/html/2608.28625#bib.bib2)\)substantially improves performance and narrows this gap\. Our analysis identifies embedding geometry as a key bottleneck: without regularization, predictive training tends to produce anisotropic representations whose variance is concentrated in a few dominant directions\. These findings indicate that within\-document prediction is a viable citation\-free signal when embedding geometry is carefully controlled\.

## 2Related Work

#### Scientific document representation\.

Scientific document encoders range from domain\-adapted language models such as SciBERTBeltagyet al\., \([2019](https://arxiv.org/html/2608.28625#bib.bib5)\)to document\-level retrieval models trained with cross\-document supervision\. Building on SciBERT as its foundation, SPECTERCohanet al\., \([2020](https://arxiv.org/html/2608.28625#bib.bib9)\)learns embeddings of titles and abstracts using citation\-based triplets, and SPECTER2Singhet al\., \([2023](https://arxiv.org/html/2608.28625#bib.bib21)\)extends this approach through multi\-format training with adapters\. These models are strong on citation\-derived benchmarks because their supervision is closely aligned with the evaluation signal\. Recent work such as SemCSEBrinner & Zarriess, \([2025](https://arxiv.org/html/2608.28625#bib.bib6)\)explores non\-citation semantic supervision through LLM\-generated summaries and contrastive positive pairs\. SciJEPA instead isolates a citation\-free and non\-contrastive signal from the internal discourse structure of individual papers\.

#### Document structure and facets\.

Prior work also exploits scientific document structure\. Hierarchical architectures such as HDTHeet al\., \([2024](https://arxiv.org/html/2608.28625#bib.bib14)\)improve long\-document modeling through hierarchical attention\. CoSAEmbSingh & Singh, \([2024](https://arxiv.org/html/2608.28625#bib.bib22)\)uses full\-text sections as aspect\-specific views and trains section\-aware embeddings with supervised contrastive triplet loss\. FLeWDouet al\., \([2025](https://arxiv.org/html/2608.28625#bib.bib10)\)uses citation intent and citation frequency to learn background\-, method\-, and result\-oriented representations\. These methods show that sections and facets are useful, but typically use structure for long\-document encoding, contrastive matching, or citation\-informed facet learning\. SciJEPA uses structure differently: section roles define asymmetric prediction tasks within the same paper, where title–abstract, method, and conclusion are treated as discourse components connected by directional dependencies rather than interchangeable views\.

#### Predictive learning and geometry\.

Joint\-Embedding Predictive Architectures \(JEPA\) were proposed as an alternative to reconstruction\-based and contrastive self\-supervised learningLeCunet al\., \([2022](https://arxiv.org/html/2608.28625#bib.bib17)\)\. Instead of reconstructing inputs or contrasting positive and negative pairs, JEPA predicts the latent representation of a target view from a context view\. This paradigm has been successful in image and video learningAssranet al\., \([2023](https://arxiv.org/html/2608.28625#bib.bib1)\); Bardeset al\., \([2023](https://arxiv.org/html/2608.28625#bib.bib4),[2024](https://arxiv.org/html/2608.28625#bib.bib3)\)and has recently been explored for languageHuanget al\., \([2025](https://arxiv.org/html/2608.28625#bib.bib15)\); Gillinet al\., \([2026](https://arxiv.org/html/2608.28625#bib.bib11)\)\. Because non\-contrastive predictive methods lack explicit negatives, they raise questions about collapse and anisotropy\. LeJEPABalestriero & LeCun, \([2025](https://arxiv.org/html/2608.28625#bib.bib2)\)studies scalable non\-contrastive joint\-embedding learning and introduces Sliced Isotropic Gaussian Regularization \(SIGReg\) as a way to encourage non\-collapsed and more isotropic representations\. We build on this regularization idea and study how SIGReg affects global isotropy, dominant\-component concentration, and local retrieval alignment in scientific document embeddings\.

## 3Methodology

xcx\_\{c\}contextxtx\_\{t\}targetonline encoderfθf\_\{\\theta\}target encoderfξf\_\{\\xi\}hch\_\{c\}predictorgϕg\_\{\\phi\}z^t\\hat\{z\}\_\{t\}ztz\_\{t\}ℒpred\\mathcal\{L\}\_\{\\mathrm\{pred\}\}ℒSIGReg\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}EMAℓ2\\ell\_\{2\}\+ std\.stop\-grad

Figure 1:Overview of SciJEPA for title\+abstract→\\tomethod and method→\\toconclusion\. The online encoder predicts a target representation through a predictor, while the target encoder is EMA\-updated and stop\-gradient\. SIGReg regularizes the normalized online representation before the predictor; preprocessing details are given in Appendix[B](https://arxiv.org/html/2608.28625#A2)\.### 3\.1Asymmetric Within\-document Prediction

We formulate scientific document pretraining as an*asymmetric within\-document prediction*problem\. Given a context sectionxcx\_\{c\}and a target sectionxtx\_\{t\}, the model encodes each section into a latent representation and learns to predict the target representation from the context representation\. For scientific papers, we instantiate this framework with two directional tasks:

title\+abstract→method,\\displaystyle\\rightarrow\\text\{method\},\(1\)method→conclusion\.\\displaystyle\\rightarrow\\text\{conclusion\}\.\(2\)The first task predicts methodological content from a summary\-level context, while the second predicts outcome\-oriented content from methodological information\. These pairs are not interchangeable views: their direction reflects discourse dependencies within the paper\.

### 3\.2Predictive Architecture and Regularization

Figure[1](https://arxiv.org/html/2608.28625#S3.F1)summarizes the SciJEPA architecture\. In non\-contrastive latent prediction, the loss only enforces agreement between learned representations\. Without negatives or token reconstruction, a symmetric agreement objective may collapse by mapping different inputs to identical or nearly identical embeddings\. SciJEPA therefore follows the online–target design used in non\-contrastive self\-supervised learningGrillet al\., \([2020](https://arxiv.org/html/2608.28625#bib.bib13)\); Chen & He, \([2021](https://arxiv.org/html/2608.28625#bib.bib8)\); LeCunet al\., \([2022](https://arxiv.org/html/2608.28625#bib.bib17)\); Assranet al\., \([2023](https://arxiv.org/html/2608.28625#bib.bib1)\): the online branch is optimized by back\-propagation, while the target branch provides a slowly evolving reference representation\.

Given a context sectionxcx\_\{c\}and a target sectionxtx\_\{t\}, the online encoderfθf\_\{\\theta\}produces a pooled context representationhc=fθ​\(xc\)h\_\{c\}=f\_\{\\theta\}\(x\_\{c\}\), while the target encoderfξf\_\{\\xi\}produceszt=fξ​\(xt\)z\_\{t\}=f\_\{\\xi\}\(x\_\{t\}\)\. Gradients are stopped onztz\_\{t\}, so the target representation is fixed for the current update\. The target encoder is not directly optimized by the predictive loss; instead, after each update, its parameters are updated as an exponential moving average \(EMA\) of the online encoder parameters:

ξ←m​ξ\+\(1−m\)​θ,\\xi\\leftarrow m\\xi\+\(1\-m\)\\theta,\(3\)wherem∈\[0,1\)m\\in\[0,1\)is the EMA momentum, set close to 1 so that the target encoder changes slowly\. Thus, the target branch follows the learned representation space with a delay, providing a stable reference for prediction and reducing the risk of representational collapse\.

The predictorgϕg\_\{\\phi\}maps the context representation into the target latent space through a residual transformation:

z^t=hc\+gϕ​\(hc\)\.\\hat\{z\}\_\{t\}=h\_\{c\}\+g\_\{\\phi\}\(h\_\{c\}\)\.\(4\)This predictor is needed because the context and target sections play different discourse roles\. A title–abstract representation should be related to a method representation, but it should not be forced to be identical to it\. The predictor therefore absorbs part of the transformation between section roles while preserving the online encoder output as a general document representation\.

For each training exampleii, the predictive loss matches the predicted representationz^t,i\\hat\{z\}\_\{t,i\}to the stop\-gradient target representationzt,iz\_\{t,i\}:

ℓpred\(i\)=1−cos⁡\(z^t,i,zt,i\)\.\\ell\_\{\\mathrm\{pred\}\}^\{\(i\)\}=1\-\\cos\(\\hat\{z\}\_\{t,i\},z\_\{t,i\}\)\.\(5\)The batch predictive loss isℒpred=1B​∑iℓpred\(i\)\\mathcal\{L\}\_\{\\mathrm\{pred\}\}=\\frac\{1\}\{B\}\\sum\_\{i\}\\ell\_\{\\mathrm\{pred\}\}^\{\(i\)\}\.

The online–target design stabilizes the prediction target, but it does not control how document embeddings are distributed globally\. Sinceℒpred\\mathcal\{L\}\_\{\\mathrm\{pred\}\}contains no negatives, different documents are not explicitly encouraged to occupy distinct regions of the space\. We add SIGRegBalestriero & LeCun, \([2025](https://arxiv.org/html/2608.28625#bib.bib2)\)to the batch of normalized online representations before the predictor\. SIGReg regularizes random one\-dimensional projections of the batch distribution toward a Gaussian reference, discouraging collapse and dominant\-direction concentration\.

The final training objective is

ℒ=1B​∑i=1Bℓpred\(i\)\+λ​ℒSIGReg\.\\mathcal\{L\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\ell\_\{\\mathrm\{pred\}\}^\{\(i\)\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}\.\(6\)
By default, evaluation uses the online encoder outputfθ​\(x\)f\_\{\\theta\}\(x\), which serves as the directly optimized general document representation\. For analysis, we also compare the predictor branchfθ​\(x\)\+gϕ​\(fθ​\(x\)\)f\_\{\\theta\}\(x\)\+g\_\{\\phi\}\(f\_\{\\theta\}\(x\)\)and the target branchfξ​\(x\)f\_\{\\xi\}\(x\)\.

### 3\.3Controlled Contrastive Baseline

A key question is how much performance can be obtained from the same citation\-free within\-document section pairs under a strong non\-citation baseline\. Since section\-based contrastive learning has been effective for technical\-document representation learningZuoet al\., \([2025](https://arxiv.org/html/2608.28625#bib.bib25)\), we include a controlled CL baseline with the same encoder backbone, pretraining corpus, section extraction procedure, and positive section pairs as SciJEPA\. Unlike SciJEPA, CL replaces latent prediction with InfoNCE and in\-batch negatives\. This baseline does not isolate the effect of negatives, but it provides a strong comparison under the same section\-pair supervision\.

Given two viewsxi\(1\)x\_\{i\}^\{\(1\)\}andxi\(2\)x\_\{i\}^\{\(2\)\}from the same document, the encoder produces pooled andℓ2\\ell\_\{2\}\-normalized representationshi\(1\)h\_\{i\}^\{\(1\)\}andhi\(2\)h\_\{i\}^\{\(2\)\}\. We optimize an InfoNCE loss with in\-batch negatives from other documents:

ℒcl\(i\)=−log⁡exp⁡\(sim​\(hi\(1\),hi\(2\)\)/τ\)exp⁡\(sim​\(hi\(1\),hi\(2\)\)/τ\)\+∑j≠iexp⁡\(sim​\(hi\(1\),hj\(1\)\)/τ\),\\mathcal\{L\}\_\{\\mathrm\{cl\}\}^\{\(i\)\}=\-\\log\\frac\{\\exp\\left\(\\mathrm\{sim\}\(h\_\{i\}^\{\(1\)\},h\_\{i\}^\{\(2\)\}\)/\\tau\\right\)\}\{\\exp\\left\(\\mathrm\{sim\}\(h\_\{i\}^\{\(1\)\},h\_\{i\}^\{\(2\)\}\)/\\tau\\right\)\+\\sum\_\{j\\neq i\}\\exp\\left\(\\mathrm\{sim\}\(h\_\{i\}^\{\(1\)\},h\_\{j\}^\{\(1\)\}\)/\\tau\\right\)\},\(7\)wheresim​\(⋅,⋅\)\\mathrm\{sim\}\(\\cdot,\\cdot\)is cosine similarity andτ\\tauis the temperature\. The final contrastive loss averages Equation \([7](https://arxiv.org/html/2608.28625#S3.E7)\) over the batch\. This baseline is not intended to reproduce citation\-supervised SPECTER\-style training\.

## 4Experimental Setup

### 4\.1Pretraining Data

We pretrain on S2ORC\-ArXiv111[https://huggingface\.co/datasets/AlgorithmicResearchGroup/s2orc\_arxiv](https://huggingface.co/datasets/AlgorithmicResearchGroup/s2orc_arxiv), a Hugging Face release of S2ORCLoet al\., \([2020](https://arxiv.org/html/2608.28625#bib.bib18)\), which provides structured full text for open\-access papers\. We filter papers with very short abstracts, abnormal section counts, short bodies, or malformed headings, and remove papers whosecorpus\_idappears in any evaluation benchmark\. After filtering and decontamination, we obtain 1\.14M unique retained documents\. During training, we sample section\-pair examples from this set with replacement until reaching a fixed budget of approximately 5M training examples\. Thus, 5M refers to the number of sampled training examples rather than the number of unique documents\.

For each paper, we extract title\+abstract, method, and conclusion/results sections using heading\-keyword matching with positional fallbacks\. Papers without method sections are discarded; papers without conclusions contribute only the title\+abstract→\\rightarrowmethod task\. Additional extraction details and statistics are provided in Appendix[A](https://arxiv.org/html/2608.28625#A1)\.

### 4\.2Model and Training Details

Unless otherwise stated, all models use SciBERT\(Beltagyet al\.,,[2019](https://arxiv.org/html/2608.28625#bib.bib5)\)as the encoder backbone and mean pooling over non\-padding tokens\. The predictor is a two\-layer MLP with architecture ofLayerNorm​\(d\)→Linear​\(d,dp\)→GELU→Linear​\(dp,d\)\\mathrm\{LayerNorm\}\(d\)\\rightarrow\\mathrm\{Linear\}\(d,d\_\{p\}\)\\rightarrow\\mathrm\{GELU\}\\rightarrow\\mathrm\{Linear\}\(d\_\{p\},d\), whered=768d=768anddp=2048d\_\{p\}=2048\. Its output is added residually to the encoder representation, as in Equation \([4](https://arxiv.org/html/2608.28625#S3.E4)\)\. The final linear layer is zero\-initialized, so the predictor starts close to the identity map and gradually learns a target\-oriented transformation\. The target encoder uses the EMA schedule:

mt=mfinal−\(mfinal−mbase\)⋅12​\(1\+cos⁡\(π​t/T\)\),m\_\{t\}=m\_\{\\mathrm\{final\}\}\-\(m\_\{\\mathrm\{final\}\}\-m\_\{\\mathrm\{base\}\}\)\\cdot\\tfrac\{1\}\{2\}\(1\+\\cos\(\\pi t/T\)\),\(8\)withmbase=0\.996m\_\{\\mathrm\{base\}\}=0\.996andmfinal=0\.999m\_\{\\mathrm\{final\}\}=0\.999\. When enabled, SIGReg is applied to the online encoder representation before the predictor using 512 random projection slices; implementation details are described in Appendix[B](https://arxiv.org/html/2608.28625#A2)\.

We train with AdamW, weight decay0\.010\.01, encoder learning rate5×10−65\\times 10^\{\-6\}, predictor learning rate1×10−41\\times 10^\{\-4\}, cosine decay, 50 warmup steps, and gradient clipping at 1\.0\. We define one training epoch as one pass over a fixed stream of 5M section\-pair examples sampled with replacement from the pool of 1\.14M unique retained documents\. All models are trained for one such epoch on a single NVIDIA H100 GPU with per\-device batch size 512 and gradient accumulation over 8 micro\-batches \(effective batch size 4,096\)\. We use gradient checkpointing andbf16mixed precision\.

### 4\.3Baselines

We consider two groups of baselines\. First, for a controlled objective\-level comparison, we train models with the same encoder backbone, pretraining corpus, section extraction procedure, and optimization pipeline:CL\(*contrastive learning*\),SciJEPA, andSciJEPA \+ SIGReg\. The CL model replaces the predictive loss with an InfoNCE objective\. Positive pairs are constructed from the same section pairs used by SciJEPA, and negatives are drawn from other documents in the same in\-batch pool\. Since training uses a per\-device batch size of 512 with 8 gradient accumulation steps, the InfoNCE negatives are computed within each 512\-example micro\-batch, while the effective optimization batch size remains 4,096 for all controlled models\. The temperature follows a cosine ramp from 0\.05 to 0\.1 after warmup\. This baseline is intended to isolate the effect of the training objective, not to reproduce citation\-supervised SPECTER\-style training\.

Second, we compare against publicly available scientific\-document encoders used off the shelf:SciBERTwith mean pooling\(Beltagyet al\.,,[2019](https://arxiv.org/html/2608.28625#bib.bib5)\),SPECTER\(Cohanet al\.,,[2020](https://arxiv.org/html/2608.28625#bib.bib9)\), andSPECTER2\(Singhet al\.,,[2023](https://arxiv.org/html/2608.28625#bib.bib21)\)\.222We use the public Hugging Face checkpoints:[https://huggingface\.co/allenai/scibert\_scivocab\_uncased](https://huggingface.co/allenai/scibert_scivocab_uncased),[https://huggingface\.co/allenai/specter](https://huggingface.co/allenai/specter), and[https://huggingface\.co/allenai/specter2](https://huggingface.co/allenai/specter2)\.For SPECTER2, we use the benchmark\-specific adapter recommended by the SciRepEval setup\.

### 4\.4Evaluation

Table 1:Evaluation protocols and relevance signals\. RELISH and high\-influence citation require ranking among curated or citation\-related candidates and are relatively fine\-grained, whereas cite prediction is a coarser triplet discrimination task\. SciDocs tasks rank task\-specific candidate sets using citation\-, co\-citation\-, co\-view\-, or co\-read\-based relevance\.We evaluate on scientific retrieval benchmarks that differ in both relevance signal and candidate granularity: RELISHBrown & Zhou, \([2019](https://arxiv.org/html/2608.28625#bib.bib7)\), high\-influence citationSinghet al\., \([2023](https://arxiv.org/html/2608.28625#bib.bib21)\), SciDocsCohanet al\., \([2020](https://arxiv.org/html/2608.28625#bib.bib9)\)\(*cite*,*co\-cite*,*co\-view*, and*co\-read*\), and cite predictionCohanet al\., \([2020](https://arxiv.org/html/2608.28625#bib.bib9)\)\. We report NDCG for RELISH, MAP for high\-influence citation, MAP and NDCG for SciDocs, and triplet accuracy for cite prediction\. Unless otherwise stated, we use SPECTER and SPECTER2 embeddings withoutℓ2\\ell\_\{2\}\-normalization, following their original setup\. For SciBERT and our trained models, weℓ2\\ell\_\{2\}\-normalize embeddings before evaluation\. Table[1](https://arxiv.org/html/2608.28625#S4.T1)summarizes the evaluation protocols\.

These protocol differences matter for interpretation\. First, several benchmarks are directly citation\-aligned, so citation\-supervised encoders such as SPECTER and SPECTER2 are evaluated under signals close to their training objective\. SciJEPA, by contrast, never observes citation links during pretraining; these tasks therefore test whether within\-document discourse prediction transfers to cross\-document retrieval\. Second, candidate granularity differs across tasks: RELISH and high\-influence citation require fine\-grained ranking among relatively related candidates, while cite prediction tests coarser positive–negative discrimination\. This helps interpret if models show different performance patterns across benchmarks\.

#### Geometry diagnostics\.

In addition to retrieval metrics, we analyze embedding geometry using alignment and uniformityWang & Isola, \([2020](https://arxiv.org/html/2608.28625#bib.bib23)\), and singular spectrum deviation \(SSD\), using the dimension\-normalized definition ofZuoet al\., \([2025](https://arxiv.org/html/2608.28625#bib.bib25)\), motivated by prior analyses of anisotropy through singular\-value spectraGodeyet al\., \([2024](https://arxiv.org/html/2608.28625#bib.bib12)\)\. Alignment measures the distance between related document pairs; lower values indicate that positives are closer\. Uniformity measures how evenly embeddings spread over the hypersphere; lower values indicate better global spread\. SSD measures deviation of the dimension\-normalized singular\-value spectrum from an isotropic reference; lower values indicate less dimensional concentration or anisotropy\. These diagnostics are not retrieval objectives, but help characterize how the embedding space balances local matching and global geometric regularity\.

We compute these diagnostics on a fixed sample of 2,000 cite\-prediction triplets\. Uniformity and SSD are computed over all 6,000 query, positive, and negative embeddings, while alignment is computed over the 2,000 query–positive pairs\.

## 5Results

### 5\.1Main Results

Table 2:Main results on RELISH, high\-influence citation, and cite prediction\.Table 3:Main results on the four SciDocs retrieval tasks\.Tables[2](https://arxiv.org/html/2608.28625#S5.T2)and[3](https://arxiv.org/html/2608.28625#S5.T3)compare SciJEPA with controlled and off\-the\-shelf baselines\. Plain SciJEPA improves substantially over the SciBERT backbone on all benchmarks, showing that asymmetric within\-document prediction provides a useful citation\-free training signal\. However, it remains below the controlled CL baseline in most tasks, indicating that predictive training is currently less robust than contrastive learning with in\-batch negatives under the same section\-pair supervision\.

Adding SIGReg improves SciJEPA across all tasks and substantially narrows the gap to CL\. The strongest result appears on high\-influence citation, where SciJEPA \+0\.0025⋅0\.0025\\cdotSIGReg reaches 44\.06 MAP, slightly outperforming CL and ranking close to SPECTER2 \+ Adapters\. This suggests that geometric regularization is a central ingredient for making predictive within\-document learning competitive\.

Compared with off\-the\-shelf scientific encoders, SciJEPA remains competitive but does not surpass the strongest citation\-supervised baselines\. SPECTER2 with task\-specific adapters achieves the best results on most tasks, while SPECTER remains strongest on cite prediction and SciDocs\-Cite MAP\. This is expected given the citation alignment of many evaluation benchmarks333Prior workOstendorffet al\., \([2022](https://arxiv.org/html/2608.28625#bib.bib19)\)reports metadata overlap between SPECTER’s training corpus and SciDocs evaluation papers, though not with SciDocs gold labels\.and reinforces our main interpretation: within\-document prediction is a viable citation\-free signal, but citation\-supervised encoders retain an advantage on citation\-derived tasks\.

The optimal SIGReg weight is task\-dependent: stronger regularization helps RELISH and high\-influence citation, whereas smaller values work better for cite prediction and SciDocs\. We analyze this trade\-off in the next section\.

### 5\.2Effect of SIGReg

![Refer to caption](https://arxiv.org/html/2608.28625v1/sigreg_ablation_all.png)Figure 2:Effect of the SIGReg weightλ\\lambdaon downstream performance and embedding geometry\. Moderate regularization improves several retrieval benchmarks, especially RELISH and high\-influence citation\. Largerλ\\lambdagenerally reduces spectral concentration but can weaken alignment and hurt some task metrics, revealing a trade\-off between global geometric regularity and local similarity preservation\.To study geometric regularization, we vary the SIGReg weightλ\\lambdaand evaluate both downstream performance and the geometry diagnostics defined in Section[4\.4](https://arxiv.org/html/2608.28625#S4.SS4): alignment, uniformity, and SSD \(Figure[2](https://arxiv.org/html/2608.28625#S5.F2)\)\. The effect of SIGReg is non\-monotonic and task\-dependent: RELISH peaks atλ=0\.005\\lambda=0\.005, high\-influence citation atλ=0\.0025\\lambda=0\.0025, while cite prediction and SciDocs favor smaller regularization\.

A plausible explanation is that these tasks differ in candidate granularity\. RELISH and high\-influence citation require fine\-grained ranking among curated or citation\-related candidates\. In this setting, reducing dominant\-direction effects can help: if unregularized SciJEPA concentrates variance in a few global directions, subtle differences among already related papers may be compressed\. Moderate SIGReg spreads variance more evenly across dimensions and can improve ranking resolution\.

The geometry diagnostics support this view\. Asλ\\lambdaincreases, SSD decreases, indicating less concentration in a few singular directions\. However, stronger SIGReg tends to weaken local alignment beyond the moderate range\. This helps explain why cite prediction and SciDocs, which rely on keeping citation\- or usage\-related positives close to separate them from broader candidate sets, prefer smallerλ\\lambda\. One interpretation is that SIGReg increases the effective dimensionality of the embedding space: this helps when dominant directions compress fine\-grained distinctions, but excessive regularization may spread variance into weakly task\-relevant directions\. In distance\-based retrieval, these extra directions can act as noise, increasing pairwise distances and weakening positive\-pair alignment\. The bestλ\\lambdatherefore depends on whether a benchmark benefits more from fine\-grained ranking resolution or from compact local neighborhoods\.

### 5\.3Analysis of Encoding Branches

![Refer to caption](https://arxiv.org/html/2608.28625v1/encoder_comparison.png)Figure 3:Comparison of online encoder, predictor, and target encoder representations across SIGReg weights\. The online encoder performs best on fine\-grained ranking tasks, while the predictor output is more robust on cite prediction and average SciDocs retrieval\. The target encoder remains consistently weaker\.SciJEPA exposes three possible inference representations: the online encoder, the predictor output, and the EMA target encoder\. All our main results use the online encoder by default, but Figure[3](https://arxiv.org/html/2608.28625#S5.F3)shows that the branches behave differently across tasks\. On fine\-grained ranking benchmarks such as RELISH and high\-influence citation, the online encoder performs best\. By contrast, on cite prediction and average SciDocs retrieval, the predictor output outperforms the online encoder across a broad range ofλ\\lambda, while the target encoder remains consistently weakest\.

This suggests that the predictor is not merely a disposable training head\. The online encoder appears to preserve fine\-grained local distinctions, which helps when candidates are already related and must be carefully ranked\. The predictor output, however, applies a target\-oriented transformation learned for section prediction; this may smooth or reweight the representation in a way that is less sensitive to stronger SIGReg and more suitable for coarser citation\-style discrimination\. The target encoder is useful as a stable EMA training target, but its lagged parameters make it less effective as an inference representation\.

## 6Conclusion

We introduced SciJEPA, a citation\-free predictive pretraining framework for scientific document representation based on asymmetric within\-document section prediction\. SciJEPA learns useful representations without citation links, but plain predictive training remains weaker than a controlled contrastive baseline using the same section\-pair supervision\.

Our results show that embedding geometry is central to making predictive learning effective: SIGReg improves SciJEPA, but its effect is task\-dependent and overly strong regularization can weaken local alignment\. Branch analysis further suggests that different parts of the predictive architecture support different retrieval regimes\. Overall, within\-document predictive learning is a promising citation\-free complement to citation\-supervised scientific document representation\.

## Acknowledgments

We thank the anonymous reviewers for their insightful comments, the CLEPS infrastructure at Inria Paris for computational resources, and Younes Djemmal, Kim Gerdes, and Kirian Guiller for their helpful discussions and feedback\.

This work was partly funded by the last author’s chair in the PRAIRIE institute funded by the French national agency ANR as part of the “Investissements d’avenir” programme under the reference ANR\-19\-P3IA\-0001\.

- Assranet al\., \(2023\)Assran M\., Duval Q\., Misra I\., Bojanowski P\., Vincent P\., Rabbat M\., LeCun Y\. & Ballas N\.\(2023\)\.Self\-supervised learning from images with a joint\-embedding predictive architecture\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, p\. 15619–15629\.
- Balestriero & LeCun, \(2025\)Balestriero R\. & LeCun Y\.\(2025\)\.Lejepa: Provable and scalable self\-supervised learning without the heuristics\.arXiv preprint arXiv:2511\.08544\.
- Bardeset al\., \(2024\)Bardes A\., Garrido Q\., Ponce J\., Chen X\., Rabbat M\., LeCun Y\., Assran M\. & Ballas N\.\(2024\)\.Revisiting feature prediction for learning visual representations from video\.arXiv preprint arXiv:2404\.08471\.
- Bardeset al\., \(2023\)Bardes A\., Ponce J\. & LeCun Y\.\(2023\)\.Mc\-jepa: A joint\-embedding predictive architecture for self\-supervised learning of motion and content features\.arXiv preprint arXiv:2307\.12698\.
- Beltagyet al\., \(2019\)Beltagy I\., Lo K\. & Cohan A\.\(2019\)\.Scibert: A pretrained language model for scientific text\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\), p\. 3615–3620\.
- Brinner & Zarriess, \(2025\)Brinner M\. F\. & Zarriess S\.\(2025\)\.Semcse: Semantic contrastive sentence embeddings using llm\-generated summaries for scientific abstracts\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p\. 32692–32707\.
- Brown & Zhou, \(2019\)Brown P\. & Zhou Y\.\(2019\)\.Large expert\-curated database for benchmarking document similarity detection in biomedical literature search\.Database,2019, baz085\.
- Chen & He, \(2021\)Chen X\. & He K\.\(2021\)\.Exploring simple siamese representation learning\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, p\. 15750–15758\.
- Cohanet al\., \(2020\)Cohan A\., Feldman S\., Beltagy I\., Downey D\. & Weld D\. S\.\(2020\)\.Specter: Document\-level representation learning using citation\-informed transformers\.InProceedings of the 58th annual meeting of the association for computational linguistics, p\. 2270–2282\.
- Douet al\., \(2025\)Dou Z\., Wang D\., Zhuang F\., Ren J\. & Hu Y\.\(2025\)\.Flew: Facet\-level and adaptive weighted representation learning of scientific documents\.arXiv preprint arXiv:2509\.07531\.
- Gillinet al\., \(2026\)Gillin T\., Lalani A\., Zhang K\. & Salles M\. M\.\(2026\)\.Bert\-jepa: Reorganizing cls embeddings for language\-invariant semantics\.arXiv preprint arXiv:2601\.00366\.
- Godeyet al\., \(2024\)Godey N\., de la Clergerie É\. & Sagot B\.\(2024\)\.Why do small language models underperform? studying language model saturation via the softmax bottleneck\.arXiv preprint arXiv:2404\.07647\.
- Grillet al\., \(2020\)Grill J\.\-B\., Strub F\., Altché F\., Tallec C\., Richemond P\., Buchatskaya E\., Doersch C\., Avila Pires B\., Guo Z\., Gheshlaghi Azar M\.et al\.\(2020\)\.Bootstrap your own latent\-a new approach to self\-supervised learning\.Advances in neural information processing systems,33, 21271–21284\.
- Heet al\., \(2024\)He H\., Flicke M\., Buchmann J\., Gurevych I\. & Geiger A\.\(2024\)\.Hdt: Hierarchical document transformer\.arXiv preprint arXiv:2407\.08330\.
- Huanget al\., \(2025\)Huang H\., LeCun Y\. & Balestriero R\.\(2025\)\.Llm\-jepa: Large language models meet joint embedding predictive architectures\.arXiv preprint arXiv:2509\.14252\.
- Jurgenset al\., \(2018\)Jurgens D\., Kumar S\., Hoover R\., McFarland D\. & Jurafsky D\.\(2018\)\.Measuring the evolution of a scientific field through citation frames\.Transactions of the Association for Computational Linguistics,6, 391–406\.doi:[10\.1162/tacl\_a\_00028](https://doi.org/10.1162/tacl_a_00028)\.
- LeCunet al\., \(2022\)LeCun Y\.et al\.\(2022\)\.A path towards autonomous machine intelligence version 0\.9\. 2, 2022\-06\-27\.Open Review,62\(1\), 1–62\.
- Loet al\., \(2020\)Lo K\., Wang L\. L\., Neumann M\., Kinney R\. & Weld D\.\(2020\)\.S2ORC: The semantic scholar open research corpus\.InD\. Jurafsky, J\. Chai, N\. Schluter & J\. Tetreault, Éds\.,Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p\. 4969–4983, Online: Association for Computational Linguistics\.doi:[10\.18653/v1/2020\.acl\-main\.447](https://doi.org/10.18653/v1/2020.acl-main.447)\.
- Ostendorffet al\., \(2022\)Ostendorff M\., Rethmeier N\., Augenstein I\., Gipp B\. & Rehm G\.\(2022\)\.Neighborhood contrastive learning for scientific document representations with citation embeddings\.InY\. Goldberg, Z\. Kozareva & Y\. Zhang, Éds\.,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p\. 11670–11688, Abu Dhabi, United Arab Emirates: Association for Computational Linguistics\.doi:[10\.18653/v1/2022\.emnlp\-main\.802](https://doi.org/10.18653/v1/2022.emnlp-main.802)\.
- Radicchiet al\., \(2008\)Radicchi F\., Fortunato S\. & Castellano C\.\(2008\)\.Universality of citation distributions: Toward an objective measure of scientific impact\.Proceedings of the National Academy of Sciences,105\(45\), 17268–17272\.
- Singhet al\., \(2023\)Singh A\., D’Arcy M\., Cohan A\., Downey D\. & Feldman S\.\(2023\)\.Scirepeval: A multi\-format benchmark for scientific document representations\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p\. 5548–5566\.
- Singh & Singh, \(2024\)Singh S\. & Singh M\.\(2024\)\.Cosaemb: Contrastive section\-aware aspect embeddings for scientific articles\.InProceedings of the Fourth Workshop on Scholarly Document Processing \(SDP 2024\), p\. 283–292\.
- Wang & Isola, \(2020\)Wang T\. & Isola P\.\(2020\)\.Understanding contrastive representation learning through alignment and uniformity on the hypersphere\.InInternational conference on machine learning, p\. 9929–9939: PMLR\.
- Xinget al\., \(2021\)Xing Y\., Wang F\., Zeng A\. & Ying F\.\(2021\)\.Solving the cold\-start problem in scientific credit allocation\.Journal of Informetrics,15\(3\), 101157\.
- Zuoet al\., \(2025\)Zuo Y\., Gerdes K\., de La Clergerie E\. V\. & Sagot B\.\(2025\)\.Patent representation learning via self\-supervision\.arXiv preprint arXiv:2511\.10657\.

## Appendix ASection Extraction Details

We extract discourse segments from the structured section text provided by S2ORC\-ArXiv\. Each paper is represented as a sequence of section titles and section bodies\. We construct three segments: title\+abstract, method, and conclusion/results\.

#### Filtering\.

To reduce parsing noise, we discard papers with abstracts shorter than 50 characters, fewer than 3 or more than 80 sections, body text shorter than 1,000 characters, or more than 30% malformed section titles\. A section title is considered malformed by a simple heuristic if it satisfies any of the following: \(i\) title length\>120\>120characters, \(ii\) more than four periods, or \(iii\) it contains sentence\-like phrases such asif and only if,we have that,it follows, orthis holds\. These patterns capture PDF parsing artifacts where running text is mistakenly parsed as a heading\. We also remove all papers whosecorpus\_idappears in any evaluation benchmark\.

#### Method section\.

We identify method sections by matching section headings against a regex that covers method\-related terms \(e\.g\.,method,methodology,approach,framework,model architecture,technical approach,problem formulation\)\. If no heading match is found, we use two fallbacks: \(i\) select the first substantial section \(content length≥50\\geq 50characters\) after anintroductionheading; \(ii\) if no explicit introduction is found, select the second substantial section in the paper\. Papers for which no substantial method\-like section can be extracted are discarded\.

#### Conclusion/results section\.

We identify conclusion/result\-oriented sections after the selected method section by heading match \(e\.g\.,conclusion,conclusions,results,results and discussion,discussion,summary,findings\)\. If no heading match is found, we use the last substantial section after the method section as a fallback\. If no such section exists, the paper is retained but only contributes the title\+abstract→\\rightarrowmethod prediction task\.

#### Task construction\.

For papers with both method and conclusion/results sections, we construct two possible prediction tasks:

title\+abstract→method,method→conclusion/results\.\\text\{title\+abstract\}\\rightarrow\\text\{method\},\\qquad\\text\{method\}\\rightarrow\\text\{conclusion/results\}\.During training, one of the available tasks is sampled for each example\. If both tasks are available, they are sampled uniformly; otherwise, only the title\+abstract→\\rightarrowmethod task is used\.

#### Extraction statistics\.

After filtering, decontamination, and section extraction, we obtain 1\.14M unique retained documents\. During training, we sample with replacement from this pool until reaching a fixed budget of 5,000,000 section\-pair training examples\. Decontamination excludes 495,096 evaluationcorpus\_ids under the current evaluation\-task set\. In an automatic audit on 200,000 filtered/decontaminated streamed examples, 198,094 retained papers produced a valid method target \(100% of retained by construction\), and 98\.50% of retained papers produced a non\-empty conclusion/results target\. Under the training\-time sampling rule, this implies expected task proportions of 50\.75% for title\+abstract→\\rightarrowmethod and 49\.25% for method→\\rightarrowconclusion/results\.

## Appendix BSIGReg Implementation Details

We apply Sliced Isotropic Gaussian Regularization \(SIGReg\) to the online encoder representation before the predictor\. LetH∈ℝB×dH\\in\\mathbb\{R\}^\{B\\times d\}be a batch of online representations\. We first normalize each representation to unit norm,

h~i=hi‖hi‖2,\\tilde\{h\}\_\{i\}=\\frac\{h\_\{i\}\}\{\\\|h\_\{i\}\\\|\_\{2\}\},and then standardize each dimension within the batch,

zi​j=h~i​j−μjσj\+ϵ,z\_\{ij\}=\\frac\{\\tilde\{h\}\_\{ij\}\-\\mu\_\{j\}\}\{\\sigma\_\{j\}\+\\epsilon\},whereμj\\mu\_\{j\}andσj\\sigma\_\{j\}are the batch mean and standard deviation of dimensionjj\. We denote the resulting batch byZ∈ℝB×dZ\\in\\mathbb\{R\}^\{B\\times d\}\.

We sampleKKrandom unit directionsa1,…,aK∈𝕊d−1a\_\{1\},\\ldots,a\_\{K\}\\in\\mathbb\{S\}^\{d\-1\}and compute projections

pi​k=⟨zi,ak⟩\.p\_\{ik\}=\\langle z\_\{i\},a\_\{k\}\\rangle\.For each projection direction, SIGReg compares the empirical characteristic function of\{pi​k\}i=1B\\\{p\_\{ik\}\\\}\_\{i=1\}^\{B\}to the characteristic function of a standard normal distribution,

φ𝒩​\(0,1\)​\(t\)=exp⁡\(−t2/2\)\.\\varphi\_\{\\mathcal\{N\}\(0,1\)\}\(t\)=\\exp\(\-t^\{2\}/2\)\.Following SIGRegBalestriero & LeCun, \([2025](https://arxiv.org/html/2608.28625#bib.bib2)\), we use an Epps–Pulley\-style characteristic\-function statistic, which provides a differentiable discrepancy between the empirical one\-dimensional projection distribution and a standard Gaussian reference:

ℒSIGReg=1K​∑k=1KB​∑ℓ=1Lwℓ​\[\(1B​∑i=1Bcos⁡\(tℓ​pi​k\)−e−tℓ2/2\)2\+\(1B​∑i=1Bsin⁡\(tℓ​pi​k\)\)2\]\.\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}B\\sum\_\{\\ell=1\}^\{L\}w\_\{\\ell\}\\left\[\\left\(\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\cos\(t\_\{\\ell\}p\_\{ik\}\)\-e^\{\-t\_\{\\ell\}^\{2\}/2\}\\right\)^\{2\}\+\\left\(\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\sin\(t\_\{\\ell\}p\_\{ik\}\)\\right\)^\{2\}\\right\]\.whereKKis the number of random projection slices,t1,…,tLt\_\{1\},\\ldots,t\_\{L\}are the characteristic\-function evaluation points, andwℓw\_\{\\ell\}are numerical integration weights\. In our experiments, we useK=512K=512,L=17L=17, and uniformly spacedtℓ∈\[0,3\]t\_\{\\ell\}\\in\[0,3\], with trapezoidal weights multiplied by the Gaussian characteristic\-function weight\.

#### Interpretation\.

The Gaussian reference in this test is the one\-dimensional standard normal distribution\. Since embeddings are firstℓ2\\ell\_\{2\}\-normalized, SIGReg should not be interpreted as matching an unconstrained Gaussian distribution inℝd\\mathbb\{R\}^\{d\}\. A useful idealized reference is the uniform distribution on the sphere: ifu∼Unif​\(𝕊d−1\)u\\sim\\mathrm\{Unif\}\(\\mathbb\{S\}^\{d\-1\}\), then for any unit vectoraa,

d​⟨a,u⟩→𝑑𝒩​\(0,1\)as​d→∞\.\\sqrt\{d\}\\,\\langle a,u\\rangle\\xrightarrow\{d\}\\mathcal\{N\}\(0,1\)\\qquad\\text\{as \}d\\to\\infty\.Thus, Gaussian random\-projection matching after normalization is consistent with encouraging sphere\-like spread in high dimension\. However, our additional dimension\-wise standardization is not an exact spherical\-uniformity test\. It acts as an empirical scale calibration for the unit\-variance Gaussian reference and also equalizes coordinate\-wise variances\. We therefore interpret normalized and standardized SIGReg as discouraging collapsed, highly anisotropic, or dimensionally concentrated embeddings, rather than as exactly enforcing uniformity on the sphere\.

A more principled regularizer for normalized embeddings could directly match the projection distribution of the uniform sphere, or explicitly balance spherical spread with local alignment; we leave this direction for future work\.