AC-JEPA：用于视觉到符号转换的锚定式跨模态联合嵌入预测架构

# AC-JEPA: Anchored Cross-Modal Joint-Embedding Predictive Architecture for Visual-to-Symbolic Translation
Source: [https://arxiv.org/html/2610.36493](https://arxiv.org/html/2610.36493)
Chaofei HONG\*, Ziyang Chen, Baochen Wang, Christina Yi Jin, Qing Zhang

###### Abstract

Cross\-modal translation requires models to capture shared semantics across modalities while preserving the fine\-grained structure required by the target modality\. Yet in real\-world scientific domains, paired data are often imperfect, while obtaining clean pairs at scale is costly\. Supervised fine\-tuning \(SFT\) directly optimizes cross\-modal translation through categorical token\-level supervision, but this objective is sensitive to errors in the target sequences\. JEPA offers a complementary representation\-space objective, but most existing formulations focus on prediction within a modality rather than translation across modalities\. We introduce Anchored Cross\-Modal JEPA \(AC\-JEPA\), an SFT\-compatible plug\-in that predicts masked intermediate states in both cross\-modal directions using paired data\. The joint\-embedding objective targets cross\-modal structure predictable from context, while the causal SFT path retains direct token\-level supervision for the target sequence\. During self\-training, AC\-JEPA learns from continuous latent targets derived from imperfect pseudo\-labeled pairs, providing a route for using these pairs without promoting pseudo\-tokens to hard categorical supervision\. Using optical chemical structure recognition as a testbed, gold\-pair AC\-JEPA achieves 76\.07% accuracy on WildMol Simple, compared with 67\.40% for Base SFT\. Both models share complete MolParser\-7M synthetic pretraining and Real\-SFT fine\-tuning; AC\-JEPA reuses the already\-seen synthetic pairs through latent prediction without adding unique gold\-pair data\. Using the same pseudo\-labeled data for both approaches, AC\-JEPA\-based self\-training further raises accuracy to 80\.23%, whereas accuracy under direct SFT drops from 67\.40% to 63\.47%\. These results support SFT\-anchored joint\-embedding prediction as an effective strategy for cross\-modal translation under both curated and imperfect supervision\.

## 1Introduction

Scientific images encode knowledge through the identities, positions, and relations of graphical elements rather than through appearance alone\. Translating such images into symbolic or programming languages therefore requires a model to recognize local primitives, recover spatial or topological structure, resolve long\-range dependencies, and serialize the result under a strict target grammar\. This differs from coarse semantic recognition or open\-ended captioning: a single misplaced symbol can alter the encoded object or invalidate the output\. The problem is further complicated by real scientific data, where visual observations and paired sequences can be imperfect and expensive to curate at scale\. Optical Chemical Structure Recognition \(OCSR\) exemplifies these demands\. Molecular diagrams encode atoms, bonds, stereochemistry, and topology through a spatial notation, whereas SMILES represents the same molecular structure as a compact, grammar\-sensitive sequence\[[23](https://arxiv.org/html/2610.36493#bib.bib3),[8](https://arxiv.org/html/2610.36493#bib.bib7)\]\.

Autoregressive multimodal generators provide a direct solution to this conversion problem\. A visual encoder extracts image evidence, and a causal language decoder combines it with the previously generated symbols to predict the target sequence\. Supervised fine\-tuning \(SFT\) supplies exact token\-level supervision and is therefore effective for learning the target grammar and decoding procedure\[[8](https://arxiv.org/html/2610.36493#bib.bib7)\]\. Its categorical objective, however, specifies the desired output tokens without directly requiring the fused intermediate states to recover predictable structure removed from either modality\. This distinction becomes important when training uses model\-generated pairs\. Direct pseudo\-label SFT promotes every generated token to a hard target, so each local pseudo\-label error directly biases the decoder’s learned conditional distribution\[[9](https://arxiv.org/html/2610.36493#bib.bib26),[20](https://arxiv.org/html/2610.36493#bib.bib19)\]\.

The usefulness of paired data also depends on the stage and objective through which they are used\. Large synthetic corpora provide broad molecular coverage for pretraining, whereas real\-image fine\-tuning addresses the visual distribution of papers and patents\[[8](https://arxiv.org/html/2610.36493#bib.bib7)\]\. After this adaptation, simply resuming token supervision on synthetic pairs need not improve real\-scene recognition: in our small\-scale implementation, mixing MolParser\-7M synthetic pairs with Real\-SFT did not improve over Real\-SFT\-only adaptation and was more prone to overfitting\. This observation motivates extracting further structural constraints from already\-seen synthetic pairs while preserving the real\-data token anchor\.

Representation\-learning objectives such as contrastive alignment, masked reconstruction, and latent prediction provide complementary ways to learn correspondences and contextual structure across paired modalities\[[18](https://arxiv.org/html/2610.36493#bib.bib21),[10](https://arxiv.org/html/2610.36493#bib.bib20),[1](https://arxiv.org/html/2610.36493#bib.bib1)\]\. JEPA is particularly relevant because it predicts representations, focusing the objective on structure predictable from context rather than every input detail\. Most JEPA formulations, however, optimize encoders or embedding predictors; exact autoregressive translation between modalities is absent or remains separate from latent pretraining\. Applying latent prediction to a symbolic generator consequently requires more than adding a standalone pretraining loss: the predicted states must remain useful to the causal decoder, use evidence from both modalities, and avoid representational collapse\.

We address these requirements with Anchored Cross\-Modal JEPA \(AC\-JEPA\), an SFT\-compatible latent predictive objective inside the same generator used for inference\.[Figure1](https://arxiv.org/html/2610.36493#S1.F1)summarizes how the SFT anchor and latent objective interact\.

![Refer to caption](https://arxiv.org/html/2610.36493v1/figures/figure1_macro_flow_3.jpg)Figure 1:Macro view of AC\-JEPA\. Paired image and SMILES views are processed by one image\-to\-SMILES VLM, while SFT anchors causal generation and the JEPA objective regularizes intermediate hidden states through latent MSE and SIGReg\. The latent self\-training loop is shown only as an abstract training extension; data\-source and curriculum details are separated from this core mechanism view\.Autoregressive SFT learns exact translation from reliable pairs, while the latent branch extracts predictive constraints from paired data, including synthetic examples already used during pretraining\. The latent branch masks coherent image regions and symbolic spans, then predicts the corresponding intermediate states from the remaining multimodal context so that evidence from either modality can support the other\. Both branches update the same backbone, allowing latent MSE to shape the representations used for causal decoding while modality\-wise SIGReg prevents collapse\[[5](https://arxiv.org/html/2610.36493#bib.bib12)\]\. In the gold\-pair setting, this provides a distinct use for the synthetic pretraining corpus after real\-image adaptation: its pairs supervise latent recovery while Real\-SFT continues to supervise output tokens\.

AC\-JEPA further provides a noise\-tolerant route for self\-training with imperfect pseudo\-labeled pairs\. Instead of promoting model\-generated tokens to exact targets, it learns cross\-modal structure that remains predictable from the paired image and sequence, thereby refining the shared representation without converting local pseudo\-label errors into categorical supervision\.

We evaluate AC\-JEPA on OCSR as a stringent instance of continuous\-to\-symbolic translation\. Under a shared exact\-match protocol, clean\-pair training improves both WildMol Simple and Markush over the SFT anchor\. The SFT anchor and gold\-pair AC\-JEPA share complete MolParser\-7M synthetic pretraining followed by Real\-SFT adaptation, so their comparison concerns reuse of existing pairs through latent prediction rather than access to additional unique data\. Using the same model\-generated pairs, latent self\-training provides further gains, whereas direct pseudo\-label SFT degrades\. These results support SFT\-anchored latent prediction as an effective training strategy for exact cross\-modal translation with both curated and imperfect pairs\.

Our contributions are:

1. 1\.We introduce AC\-JEPA, an SFT\-compatible framework integrating cross\-modal latent prediction with causal decoding for exact visual\-to\-symbolic translation\.
2. 2\.We develop a bidirectional cross\-modal latent\-learning mechanism that captures structure predictable from multimodal context while maintaining non\-degenerate representations within each modality\.
3. 3\.We derive local bounds that relate decoder\-relevant latent residuals to next\-token decision stability and characterize clean\-output stability under pseudo\-label perturbations\.
4. 4\.We validate AC\-JEPA on OCSR, showing gains over SFT with clean pairs and over direct pseudo\-label SFT during self\-training\.

## 2Related Work

##### Multimodal representation learning\.

Multimodal representation learning seeks to capture content shared across paired modalities while retaining the fine\-grained information required for modality\-specific prediction\. Contrastive methods such as CLIP and ALIGN align global image–text representations and are effective at learning factors shared across paired observations\[[18](https://arxiv.org/html/2610.36493#bib.bib21),[11](https://arxiv.org/html/2610.36493#bib.bib22)\]\. Fusion and generative models such as ALBEF and BLIP combine alignment with cross\-modal interaction or language generation, while BEiT\-3 extends masked modeling across images, text, and paired inputs\[[14](https://arxiv.org/html/2610.36493#bib.bib23),[13](https://arxiv.org/html/2610.36493#bib.bib24),[22](https://arxiv.org/html/2610.36493#bib.bib25)\]\. These objectives support transferable multimodal representations and capable generators, but exact scientific translation can additionally depend on modality\-specific local structure that is not captured by global correspondence alone\. AC\-JEPA targets this setting by applying local masked\-state prediction to modality\-indexed positions within the same fused representation used for autoregressive decoding\. SFT preserves the task\-specific output interface, while the latent objective requires the intermediate states to retain cross\-modal structure recoverable from the remaining context\.

##### Latent predictive learning and multimodal JEPA\.

Latent predictive learning moves the prediction target from the observation space to a learned representation space\. Whereas masked autoencoding reconstructs missing observations, data2vec predicts contextualized latent targets across speech, vision, and language, and I\-JEPA predicts representations of image regions from visible context\[[10](https://arxiv.org/html/2610.36493#bib.bib20),[3](https://arxiv.org/html/2610.36493#bib.bib2),[1](https://arxiv.org/html/2610.36493#bib.bib1)\]\. The abstraction induced by such objectives depends on what is masked, which context remains visible, and how the target representation is constructed\. V\-JEPA 2 extends latent prediction to video, while LeJEPA couples predictability with SIGReg\-based control of representation collapse\[[2](https://arxiv.org/html/2610.36493#bib.bib11),[5](https://arxiv.org/html/2610.36493#bib.bib12)\]\. Recent cross\-modal variants impose different predictive relations: CrossJEPA transfers from a frozen image target to a point\-cloud encoder, VL\-JEPA predicts text embeddings from visual input and a query, and MJEPA performs intra\- and cross\-modal prediction with a unified audio\-visual encoder\[[16](https://arxiv.org/html/2610.36493#bib.bib13),[7](https://arxiv.org/html/2610.36493#bib.bib14),[21](https://arxiv.org/html/2610.36493#bib.bib15)\]\. Their primary output is an encoder representation or embedding prediction\. In contrast, AC\-JEPA places bidirectional masked\-state prediction inside the same autoregressive image\-to\-SMILES generator that is trained and deployed for exact symbolic translation\. The central compatibility constraint is therefore not latent prediction alone, but preserving an intermediate state that remains useful to the upper causal decoder\.

##### Self\-training with imperfect pseudo\-labels\.

Self\-training expands supervised data by using a trained model to generate targets for unlabeled examples\[[12](https://arxiv.org/html/2610.36493#bib.bib16)\]\. Modern methods reduce confirmation errors through confidence filtering, consistency regularization, augmentation, teacher–student updates, or momentum distillation\[[19](https://arxiv.org/html/2610.36493#bib.bib17),[24](https://arxiv.org/html/2610.36493#bib.bib18),[14](https://arxiv.org/html/2610.36493#bib.bib23)\]\. Their effectiveness nevertheless depends on the structure of the teacher’s errors, since recurring mistakes can be reinforced when predictions are reused as hard supervision\[[20](https://arxiv.org/html/2610.36493#bib.bib19)\]\. This issue is particularly consequential for symbolic generation: a sequence may be syntactically plausible and locally close to the correct output while still representing a different object\. Direct pseudo\-label SFT treats such a token substitution as an exact categorical target\. AC\-JEPA instead uses the pseudo\-sequence as part of a masked multimodal view and learns by predicting its continuous intermediate states\. Our experiments compare these two training routes using the same pseudo\-labeled data rather than attributing their difference to data selection\.

##### Optical chemical structure recognition\.

Optical Chemical Structure Recognition converts molecular diagrams from papers, patents, and scanned documents into machine\-readable molecular graphs or line notations such as SMILES\[[23](https://arxiv.org/html/2610.36493#bib.bib3)\]\. Unlike ordinary text recognition, it must recover graph topology, stereochemistry, abbreviations, drawing conventions, and, in the Markush setting, variable molecular structures\. Recent neural systems approach this problem through direct image\-to\-string generation, chemistry\-constrained graph prediction, graph classification, or large\-scale end\-to\-end recognition\[[6](https://arxiv.org/html/2610.36493#bib.bib4),[17](https://arxiv.org/html/2610.36493#bib.bib5),[15](https://arxiv.org/html/2610.36493#bib.bib6),[8](https://arxiv.org/html/2610.36493#bib.bib7)\]\. MolSight and MolRecBench\-Wild further expose the importance of noisy pretraining and generalization to real scientific documents\[[26](https://arxiv.org/html/2610.36493#bib.bib9),[25](https://arxiv.org/html/2610.36493#bib.bib10)\]\. Most progress in OCSR has focused on architectures, output representations, data scale, and decoding constraints\. Our work studies a complementary training question: how an SFT\-capable generator can learn additional cross\-modal structure from clean and pseudo\-labeled pairs through an auxiliary latent predictive objective\.

## 3Method

AC\-JEPA integrates autoregressive SFT and latent prediction within one image\-to\-SMILES VLM\. SFT trains the model through autoregressive next\-token prediction, whereas JEPA predicts latent states at an intermediate decoder layer\. Both losses update the shared backbone, so latent prediction shapes the representation used by the generator rather than a separate pretrained encoder\. A bidirectional predictor recovers masked states using the remaining multimodal context, and modality\-wise SIGReg prevents latent collapse\. The same latent objective subsequently incorporates pseudo\-labeled pairs without treating their generated SMILES as default token\-level targets\.[Figure2](https://arxiv.org/html/2610.36493#S3.F2)shows the resulting architecture and training losses\.[Sections3\.1](https://arxiv.org/html/2610.36493#S3.SS1),[3\.2](https://arxiv.org/html/2610.36493#S3.SS2),[3\.3](https://arxiv.org/html/2610.36493#S3.SS3)and[3\.4](https://arxiv.org/html/2610.36493#S3.SS4)describe the latent interface, predictive objective, training curriculum, and analysis, respectively\.

![Refer to caption](https://arxiv.org/html/2610.36493v1/figures/figure2_model_structure_imagegen_v2.jpg)Figure 2:Model structure and training losses of AC\-JEPA\. SFT supervises the causal decoder, while JEPA predicts intermediate latents under structured masking\. The clean latent is detached only as the MSE target, while SIGReg regularizes the undetached clean image and SMILES latents separately\.### 3\.1SFT\-Anchored Backbone and Latent Interface

The position of the latent interface determines whether predictive learning can affect both cross\-modal fusion and downstream decoding\. We attach it to an intermediate decoder layer after visual and symbolic context have been fused but before next\-token logits are produced\. The JEPA path projects this state for latent prediction, whereas the SFT path continues through the remaining causal decoder and language\-model head\. Both losses update the shared model below the interface, while SFT directly supervises the upper decoding path\. This arrangement keeps the latent objective inside the task\-capable generator while preserving the causal output path used at inference\.

LetIIbe a molecular image,S=\(S1,…,ST\)S=\(S\_\{1\},\\ldots,S\_\{T\}\)its gold SMILES sequence, andπ\\pithe task prompt\. The SFT anchor trains the backbonefθf\_\{\\theta\}autoregressively,

ℒSFT=−∑t=1Tlogpθ\(St∣π,I,S<t\)\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=\-\\sum\_\{t=1\}^\{T\}\\log p\_\{\\theta\}\(S\_\{t\}\\mid\\pi,I,S\_\{<t\}\)\.\(1\)This objective ties the shared representation to the grammar and causal decoding procedure used at inference\.

For a paired symbolic viewSviewS\_\{\\mathrm\{view\}\}, we construct clean and masked multimodal views

𝐕clean\\displaystyle\\mathbf\{V\}\_\{\\mathrm\{clean\}\}=\(π,I,Sview\),\\displaystyle=\(\\pi,I,S\_\{\\mathrm\{view\}\}\),\(2\)𝐕mask\\displaystyle\\mathbf\{V\}\_\{\\mathrm\{mask\}\}=m⁡\(𝐕clean,ℳimg,ℳsmi\)\.\\displaystyle=m\(\\mathbf\{V\}\_\{\\mathrm\{clean\}\};\\mathcal\{M\}\_\{\\mathrm\{img\}\},\\mathcal\{M\}\_\{\\mathrm\{smi\}\}\)\.whereℳimg\\mathcal\{M\}\_\{\\mathrm\{img\}\}andℳsmi\\mathcal\{M\}\_\{\\mathrm\{smi\}\}are the masked image and SMILES positions\. LetHθℓ​\(𝐕\)H\_\{\\theta\}^\{\\ell\}\(\\mathbf\{V\}\)be the hidden sequence at decoder layerℓ\\ell,gϕg\_\{\\phi\}a projector, andpψp\_\{\\psi\}the predictor\. The projector maps the fused hidden states into a compact space in which the masked\-view representation is trained to match its clean counterpart\. The clean latent, detached target, and masked\-view prediction are

Zclean\\displaystyle Z\_\{\\mathrm\{clean\}\}=gϕ​\(Hθℓ​\(𝐕clean\)\),\\displaystyle=g\_\{\\phi\}\(H\_\{\\theta\}^\{\\ell\}\(\\mathbf\{V\}\_\{\\mathrm\{clean\}\}\)\),Ztgt\\displaystyle Z\_\{\\mathrm\{tgt\}\}=sg⁡\(Zclean\),\\displaystyle=\\operatorname\{sg\}\(Z\_\{\\mathrm\{clean\}\}\),\(3\)Zpred\\displaystyle Z\_\{\\mathrm\{pred\}\}=pψ​\(gϕ​\(Hθℓ​\(𝐕mask\)\)\),\\displaystyle=p\_\{\\psi\}\(g\_\{\\phi\}\(H\_\{\\theta\}^\{\\ell\}\(\\mathbf\{V\}\_\{\\mathrm\{mask\}\}\)\)\),wheresg\\operatorname\{sg\}blocks predictive\-MSE gradients through the clean target\. The clean branch therefore supplies a fixed target for each prediction step, while both the masked branch and the shared backbone below the interface remain trainable\. Latent recovery is optimized only at masked positions and separately for each modality,

ℒMSE\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{MSE\}\}=∑m∈\{img,smi\}λmsem\|ℳm\|∑i∈ℳm\\displaystyle=\\sum\_\{m\\in\\\{\\mathrm\{img\},\\mathrm\{smi\}\\\}\}\\frac\{\\lambda\_\{\\mathrm\{mse\}\}^\{m\}\}\{\|\\mathcal\{M\}\_\{m\}\|\}\\sum\_\{i\\in\\mathcal\{M\}\_\{m\}\}\(4\)‖Zpred,i−Ztgt,i‖22\.\\displaystyle\\left\\\|Z\_\{\\mathrm\{pred\},i\}\-Z\_\{\\mathrm\{tgt\},i\}\\right\\\|\_\{2\}^\{2\}\.The continuous loss complements rather than replaces token\-level SFT\. SFT anchors exact generation and the output grammar, while masked latent prediction directly constrains the shared intermediate representation consumed by the decoder\.

### 3\.2Bidirectional Prediction and Latent Regulation

AC\-JEPA trains its intermediate representation by recovering masked image and symbolic states from the remaining multimodal context, while modality\-wise regulation keeps the predictive target space non\-degenerate\. We construct the recovery task by replacing contiguous SMILES spans and rectangular image\-patch blocks with modality\-specific mask tokens; prompt, padding, and special tokens remain unmasked\. Masking coherent regions removes local evidence and makes recovery rely on the surrounding structure available within and across modalities\. Bidirectional attention lets the predictor recover masked states from the full corrupted multimodal context, while the SFT generation path remains causal\.[SectionA\.1](https://arxiv.org/html/2610.36493#A1.SS1)specifies the structured masking procedure\.

Because predictive objectives can admit collapsed representations, we apply the Sketched Isotropic\-Gaussian Regularizer \(SIGReg\) to the clean latent distribution\[[5](https://arxiv.org/html/2610.36493#bib.bib12)\]\. For modalitymm, letℐm\\mathcal\{I\}\_\{m\}contain its eligible clean\-view positions in a mini\-batch and𝒵m=\{Zclean,i:i∈ℐm\}\\mathcal\{Z\}\_\{m\}=\\\{Z\_\{\\mathrm\{clean\},i\}:i\\in\\mathcal\{I\}\_\{m\}\\\}\. We regularize image and SMILES states separately,

ℒSIGReg=∑m∈\{img,smi\}λsigm​SIGReg​\(𝒵m\)\.\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}=\\sum\_\{m\\in\\\{\\mathrm\{img\},\\mathrm\{smi\}\\\}\}\\lambda\_\{\\mathrm\{sig\}\}^\{m\}\\mathrm\{SIGReg\}\(\\mathcal\{Z\}\_\{m\}\)\.\(5\)This separation prevents one modality’s variance from hiding collapse in the other\. SIGReg acts on the undetached clean latents, whereas stop\-gradient applies only to the MSE target\. Accordingly, stop\-gradient stabilizes each prediction target, whereas SIGReg constrains the representation from which the targets are formed\. The complete latent objective isℒJEPA=ℒMSE\+ℒSIGReg\\mathcal\{L\}\_\{\\mathrm\{JEPA\}\}=\\mathcal\{L\}\_\{\\mathrm\{MSE\}\}\+\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}\.

### 3\.3From SFT Anchoring to Latent Self\-Training

The training curriculum first establishes exact translation on reliable pairs and then uses the latent objective to incorporate open\-environment images without gold sequences\. This ordering is important because the JEPA branch should shape an already task\-capable representation rather than define the symbolic task from scratch\.[Figure3](https://arxiv.org/html/2610.36493#S3.F3)summarizes the four training stages\. Base SFT and all AC\-JEPA variants share domain continued pretraining \(CPT\) on the complete synthetic MolParser\-7M image–SMILES corpus, followed by autoregressive fine\-tuning on Real\-SFT 100K\[[8](https://arxiv.org/html/2610.36493#bib.bib7)\]\. Synthetic pretraining establishes broad task competence, while real\-image fine\-tuning adapts the generator to document images\. We freeze this Base SFT backbone to warm up the new JEPA modules; gold\-pair joint training and each latent self\-training run then initialize independently from the resulting checkpoint\.

![Refer to caption](https://arxiv.org/html/2610.36493v1/figures/figure3_training_curriculum_imagegen_v2.png)Figure 3:Training curriculum\. Base SFT first learns from the complete synthetic MolParser\-7M corpus and is then fine\-tuned on Real\-SFT 100K\. After JEPA\-module warm\-up with that backbone frozen, gold\-pair joint training and latent self\-training start independently from the shared checkpoint\. Gold\-pair joint training reuses the already\-seen 7M pairs for latent prediction while retaining Real\-SFT for token supervision\. Self\-training adds open\-environment images through model\-generated pseudo\-SMILES while gold\-pair SFT remains active as the anchor; pseudo\-label sets are refreshed between rounds\.Let𝒟sft\\mathcal\{D\}\_\{\\mathrm\{sft\}\}contain the reliable pairs assigned token\-level supervision and𝒟jepag\\mathcal\{D\}\_\{\\mathrm\{jepa\}\}^\{g\}contain the gold pairs used for latent prediction\. The anchored objective is

ℒanchor\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}=λsft​𝔼\(I,S\)∼𝒟sft​\[ℒSFT​\(I,S\)\]\\displaystyle=\\lambda\_\{\\mathrm\{sft\}\}\\mathbb\{E\}\_\{\(I,S\)\\sim\\mathcal\{D\}\_\{\\mathrm\{sft\}\}\}\[\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(I,S\)\]\(6\)\+𝔼\(I,S\)∼𝒟jepag​\[ℒJEPA​\(I,S\)\]\.\\displaystyle\+\\mathbb\{E\}\_\{\(I,S\)\\sim\\mathcal\{D\}\_\{\\mathrm\{jepa\}\}^\{g\}\}\[\\mathcal\{L\}\_\{\\mathrm\{JEPA\}\}\(I,S\)\]\.The two datasets may overlap but need not coincide: joint training means that both losses update the shared backbone, not that they are evaluated on the same examples\. In our gold\-pair setting,𝒟sft\\mathcal\{D\}\_\{\\mathrm\{sft\}\}is Real\-SFT and𝒟jepag\\mathcal\{D\}\_\{\\mathrm\{jepa\}\}^\{g\}reuses the synthetic MolParser\-7M pairs already used during CPT; no unique paired examples are added to the Base SFT training history\. Our small\-scale mixed\-SFT observation in[Section4\.2](https://arxiv.org/html/2610.36493#S4.SS2)motivates retaining real\-image token supervision while reusing synthetic pairs through latent prediction\. At refresh roundrr, the current model generatesS^i\(r\)\\hat\{S\}\_\{i\}^\{\(r\)\}for eachIi∈𝒟uI\_\{i\}\\in\\mathcal\{D\}\_\{u\}, forming𝒟p\(r\)=\{\(Ii,S^i\(r\)\)\}\\mathcal\{D\}\_\{p\}^\{\(r\)\}=\\\{\(I\_\{i\},\\hat\{S\}\_\{i\}^\{\(r\)\}\)\\\}\. AC\-JEPA then treats each pseudo\-SMILES as the symbolic viewSviewS\_\{\\mathrm\{view\}\}of a masked latent prediction problem rather than as default token\-level supervision:

ℒ\(r\)=ℒanchor\+ηr​𝔼\(I,S^\)∼𝒟p\(r\)​\[q⁡\(I,S^\)​ℒJEPA​\(I,S^\)\],\\mathcal\{L\}^\{\(r\)\}=\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\+\\eta\_\{r\}\\mathbb\{E\}\_\{\(I,\\hat\{S\}\)\\sim\\mathcal\{D\}\_\{p\}^\{\(r\)\}\}\[q\(I,\\hat\{S\}\)\\,\\mathcal\{L\}\_\{\\mathrm\{JEPA\}\}\(I,\\hat\{S\}\)\],\(7\)whereq⁡\(I,S^\)q\(I,\\hat\{S\}\)is a selection or reliability weight andηr\\eta\_\{r\}controls the pseudo\-labeled stream\. The supervised stream continues to supply the token\-level anchor throughout this stage\. Direct pseudo\-label SFT is retained as a comparison, so the two routes differ in how the same pseudo\-labeled data provide supervision\.[Algorithm1](https://arxiv.org/html/2610.36493#alg1)summarizes the complete training procedure\.

### 3\.4Local Analysis of Task\-Aligned Representations and Noise Tolerance

To obtain a tractable account of the proposed mechanism, we analyze a local model of AC\-JEPA under simplifying assumptions on decoder margins, subspace overlap, anchored curvature, and pseudo\-label influence\. The resulting bounds identify conditions under which lower decoder\-relevant latent residuals tighten the next\-token decision\-reversal bound and pseudo\-label perturbations remain within the clean\-output margin\. They provide a mechanism\-level interpretation rather than a general guarantee for the full nonlinear training dynamics\. Detailed proofs and further interpretation are provided in[SectionA\.5](https://arxiv.org/html/2610.36493#A1.SS5)\.

At masked SMILES positionii, letHiH\_\{i\}be the clean intermediate state,CiC\_\{i\}the corrupted multimodal context, andΔi,s−​\(Hi\)≈di,s−⊤​Hi\+ci,s−\\Delta\_\{i,s^\{\-\}\}\(H\_\{i\}\)\\approx d\_\{i,s^\{\-\}\}^\{\\top\}H\_\{i\}\+c\_\{i,s^\{\-\}\}the logit gap between the correct next tokens\+s^\{\+\}and competitors−s^\{\-\}\. Letμi,s−​\(Ci\)\>0\\mu\_\{i,s^\{\-\}\}\(C\_\{i\}\)\>0andvi,s−​\(Ci\)v\_\{i,s^\{\-\}\}\(C\_\{i\}\)be its conditional mean and variance, representing the average correct\-token margin and residual variation\. With context\-conditioned state residualUi=Hi−𝔼\[Hi∣Ci,Si\+1=s\+\]U\_\{i\}=H\_\{i\}\-\\mathbb\{E\}\[H\_\{i\}\\mid C\_\{i\},S\_\{i\+1\}=s^\{\+\}\]and latent\-interface JacobianJiJ\_\{i\}, define the decoder\-relevant residual energyRJ,i\(Ci\)=𝔼\[∥JiUi∥22∣Ci,Si\+1=s\+\]R\_\{\\mathrm\{J\},i\}\(C\_\{i\}\)=\\mathbb\{E\}\[\\\|J\_\{i\}U\_\{i\}\\\|\_\{2\}^\{2\}\\mid C\_\{i\},S\_\{i\+1\}=s^\{\+\}\]\. We assume the decoder margin direction is controlled through this interface:di,s−​di,s−⊤⪯κ​Ji⊤​Jid\_\{i,s^\{\-\}\}d\_\{i,s^\{\-\}\}^\{\\top\}\\preceq\\kappa J\_\{i\}^\{\\top\}J\_\{i\}\.

###### Theorem 3\.1\(SFT readout stability under latent residuals\)\.

LetPi,s−=Pr\[Δi,s−\(Hi\)≤0∣Ci,Si\+1=s\+\]P\_\{i,s^\{\-\}\}=\\Pr\[\\Delta\_\{i,s^\{\-\}\}\(H\_\{i\}\)\\leq 0\\mid C\_\{i\},S\_\{i\+1\}=s^\{\+\}\]\. Under the local model above,

Pi,s−\\displaystyle P\_\{i,s^\{\-\}\}≤vi,s−​\(Ci\)vi,s−​\(Ci\)\+μi,s−​\(Ci\)2\\displaystyle\\leq\\frac\{v\_\{i,s^\{\-\}\}\(C\_\{i\}\)\}\{v\_\{i,s^\{\-\}\}\(C\_\{i\}\)\+\\mu\_\{i,s^\{\-\}\}\(C\_\{i\}\)^\{2\}\}\(8\)≤κ​RJ,i​\(Ci\)κ​RJ,i​\(Ci\)\+μi,s−​\(Ci\)2\.\\displaystyle\\leq\\frac\{\\kappa R\_\{\\mathrm\{J\},i\}\(C\_\{i\}\)\}\{\\kappa R\_\{\\mathrm\{J\},i\}\(C\_\{i\}\)\+\\mu\_\{i,s^\{\-\}\}\(C\_\{i\}\)^\{2\}\}\.Thus, at a fixed positive mean margin, reducing the decoder\-relevant latent residual tightens the decision\-reversal bound\.

The symbolic MSE decomposes asℒMSEsmi=R¯Jsmi\+ℰpredsmi\\mathcal\{L\}\_\{\\mathrm\{MSE\}\}^\{\\mathrm\{smi\}\}=\\overline\{R\}\_\{\\mathrm\{J\}\}^\{\\mathrm\{smi\}\}\+\\mathcal\{E\}\_\{\\mathrm\{pred\}\}^\{\\mathrm\{smi\}\}\. After predictor warm\-up, reducing MSE beyond the initial excess risk lowers the Bayes latent residual and tightens the averaged bound under decoder–JEPA overlap and SFT margin preservation\.

To quantify pseudo\-label effects on clean outputs, considernnpseudo constraints and take as the clean reference a stationary pointθ⋆\\theta^\{\\star\}of the counterfactual anchored objective with their corresponding true sequences\{Sj\}j=1n\\\{S\_\{j\}\\\}\_\{j=1\}^\{n\}\. For pseudo constraintjj, define the latent residualrjr\_\{j\}and its perturbationeje\_\{j\}after replacingSjS\_\{j\}withS^j\\widehat\{S\}\_\{j\}:

rj​\(θ,Sj\)\\displaystyle r\_\{j\}\(\\theta;S\_\{j\}\)=Zpred,j​\(θ,Sj\)−Ztgt,j​\(θ,Sj\),\\displaystyle=Z\_\{\\mathrm\{pred\},j\}\(\\theta;S\_\{j\}\)\-Z\_\{\\mathrm\{tgt\},j\}\(\\theta;S\_\{j\}\),\(9\)ej\\displaystyle e\_\{j\}=rj​\(θ⋆,S^j\)−rj​\(θ⋆,Sj\)\.\\displaystyle=r\_\{j\}\(\\theta^\{\\star\};\\widehat\{S\}\_\{j\}\)\-r\_\{j\}\(\\theta^\{\\star\};S\_\{j\}\)\.LetAj=∂rj​\(θ,Sj\)/∂θ\|θ⋆A\_\{j\}=\\partial r\_\{j\}\(\\theta;S\_\{j\}\)/\\partial\\theta\|\_\{\\theta^\{\\star\}\}be the clean\-residual Jacobian,K≻0K\\succ 0the local anchored curvature, andλ\\lambdathe effective pseudo\-MSE weight\. Under the local shared\-Jacobian approximation, with the Jacobian\-change and SIGReg remainders made explicit in[SectionA\.5\.2](https://arxiv.org/html/2610.36493#A1.SS5.SSS2), the first\-order parameter shift is

upseudo≈−λnK−1∑j=1nAj⊤ej\.u\_\{\\mathrm\{pseudo\}\}\\approx\-\\frac\{\\lambda\}\{n\}K^\{\-1\}\\sum\_\{j=1\}^\{n\}A\_\{j\}^\{\\top\}e\_\{j\}\.\(10\)For clean contextxx, letmx,y​k\>0m\_\{x,yk\}\>0be the margin between correct tokenyyand competitorkk,dx,y​kd\_\{x,yk\}its state gradient, andJxJ\_\{x\}the parameter\-to\-state Jacobian\. Pseudo constraintjjhas signed margin influence

ζx,y​k,j=dx,y​k⊤​Jx​K−1​Aj⊤​ej\.\\zeta\_\{x,yk,j\}=d\_\{x,yk\}^\{\\top\}J\_\{x\}K^\{\-1\}A\_\{j\}^\{\\top\}e\_\{j\}\.\(11\)
###### Theorem 3\.2\(Clean\-output stability under pseudo\-label perturbations\)\.

Suppose the influences are independent with𝔼⁡\[ζx,y​k,j\]=βx,y​k\\mathbb\{E\}\[\\zeta\_\{x,yk,j\}\]=\\beta\_\{x,yk\}andVar⁡\(ζx,y​k,j\)≤σx,y​k2\\operatorname\{Var\}\(\\zeta\_\{x,yk,j\}\)\\leq\\sigma\_\{x,yk\}^\{2\}\. The first\-order margin isGx,y​klin=mx,y​k−λ​n−1​∑jζx,y​k,jG\_\{x,yk\}^\{\\mathrm\{lin\}\}=m\_\{x,yk\}\-\\lambda n^\{\-1\}\\sum\_\{j\}\\zeta\_\{x,yk,j\}\. Ifmx,y​k\>λ​βx,y​km\_\{x,yk\}\>\\lambda\\beta\_\{x,yk\}, then

Pr\[Gx,y​klin≤0\]≤λ2​σx,y​k2/nλ2​σx,y​k2/n\+\(mx,y​k−λ​βx,y​k\)2\.\\Pr\[G\_\{x,yk\}^\{\\mathrm\{lin\}\}\\leq 0\]\\leq\\frac\{\\lambda^\{2\}\\sigma\_\{x,yk\}^\{2\}/n\}\{\\lambda^\{2\}\\sigma\_\{x,yk\}^\{2\}/n\+\(m\_\{x,yk\}\-\\lambda\\beta\_\{x,yk\}\)^\{2\}\}\.\(12\)

The bound separates mean signed influenceβx,y​k\\beta\_\{x,yk\}, whose positive values consume clean margin, from variance that decreases across independent pseudo constraints\. Conditional centering𝔼⁡\[ej∣Aj\]=0\\mathbb\{E\}\[e\_\{j\}\\mid A\_\{j\}\]=0is sufficient for zero mean first\-order influence under this local model\.

## 4Experiments

### 4\.1Experimental Setup

All experiments use Qwen3\-VL\-2B\[[4](https://arxiv.org/html/2610.36493#bib.bib8)\]as the backbone and draw paired training data from UniParser/MolParser\-7M\[[8](https://arxiv.org/html/2610.36493#bib.bib7)\]\. Base SFT and all AC\-JEPA variants share the same two\-stage training history: autoregressive domain pretraining on the complete synthetic MolParser\-7M corpus, followed by fine\-tuning on Real\-SFT 100K\. We use “MolParser\-7M” below for the synthetic pretraining corpus and “Real\-SFT” for the real\-image adaptation set\. Both data sources have therefore already contributed to Base SFT; the gold\-pair comparison introduces no additional unique training pairs\. Following[Section3\.3](https://arxiv.org/html/2610.36493#S3.SS3), we warm up the newly initialized JEPA modules with this backbone frozen and then jointly optimize token\-level SFT on Real\-SFT and latent prediction on the previously seen synthetic pairs\. Latent self\-training subsequently extends the JEPA stream to pseudo\-labeled open\-environment images, while direct SFT on the same pseudo\-labels serves as the token\-level self\-training baseline\. For rounds R1 and R2, the unlabeled pool includes molecular images from USPTO\-Mol and structure figures extracted from scientific papers\. We select checkpoints on the MolParser\-7M validation split and evaluate them on the WildMol Simple and Markush sets, each containing 10,000 images collected from scientific papers and patents\[[8](https://arxiv.org/html/2610.36493#bib.bib7)\]\. Under a shared decoding and evaluation protocol, we report canonicalized exact\-match accuracy using an RDKit\-based evaluator customized for extended SMILES \(E\-SMILES\)\.

### 4\.2Results and Training Dynamics Across the Curriculum

We first evaluate how performance evolves across the AC\-JEPA training curriculum, from the Base SFT model to gold\-pair joint training and two rounds of latent self\-training\. After warming up the newly initialized JEPA modules with the backbone frozen, all three AC\-JEPA configurations start from the resulting checkpoint and are optimized independently\. Gold\-pair AC\-JEPA retains Real\-SFT as the token\-level anchor and reuses the previously seen synthetic MolParser\-7M pairs through JEPA\. For R1 and R2, the unlabeled pool includes molecular images from USPTO\-Mol and structure figures extracted from scientific papers\. Each round generates pseudo\-SMILES for this pool and adds the resulting pairs to the JEPA stream, while reliable\-pair SFT remains the task anchor\.

Table 1:Accuracy \(%\) across the AC\-JEPA curriculum on WildMol Simple and Markush\. Every row in our curriculum inherits the same MolParser\-7M pretraining and Real\-SFT fine\-tuning; gold\-pair AC\-JEPA reuses existing pairs through JEPA\. MolParser\-Base is a published reference from[Fang et al\. \[8\]](https://arxiv.org/html/2610.36493#bib.bib7), with a different architecture and optimization recipe\.[Table1](https://arxiv.org/html/2610.36493#S4.T1)summarizes performance across the proposed training curriculum\. Starting from Base SFT, joint gold\-pair training with AC\-JEPA raises accuracy by 8\.67 points on Simple and 5\.80 points on Markush\. This configuration reaches comparable Simple accuracy to MolParser\-Base and improves Markush by 4\.18 points\.

Self\-training R1 and R2 extend latent prediction with these pseudo\-pairs while retaining the SFT anchor on clean data\. R2 reaches 80\.23% on Simple and 48\.28% on Markush, exceeding the independently initialized gold\-pair configuration by 4\.16 and 6\.00 points, respectively\. The higher endpoint accuracies of both self\-training runs show that the pseudo\-pair streams benefit AC\-JEPA in this OCSR setting\.

![Refer to caption](https://arxiv.org/html/2610.36493v1/figures/figure4_multistage_training_losses.png)Figure 4:Component losses from three independent AC\-JEPA runs initialized from the same CPT\-trained, JEPA\-warmed checkpoint\. Curves show every 100 local optimization steps for gold\-pair training \(6,000 steps\), self\-training R1 \(4,500 steps\), and self\-training R2 \(4,000 steps\)\. The blocks are juxtaposed for compact comparison and are not connected across runs\. All curves report unweighted loss components\.[Figure4](https://arxiv.org/html/2610.36493#S4.F4)reports component losses for the three independent runs\. All start from the same CPT\-trained, JEPA\-warmed checkpoint with separate optimization schedules, and the juxtaposed blocks show run\-local rather than sequential steps\. Within each run, SFT and both modality\-specific MSE losses decrease; image SIGReg also decreases, while text SIGReg remains low and bounded\. These curves show joint optimization of the token\-level and latent objectives;[Table1](https://arxiv.org/html/2610.36493#S4.T1)reports endpoint accuracy\.

##### Reusing synthetic pairs after real\-image adaptation\.

The shared pretraining history is important for interpreting the gold\-pair gain: MolParser\-7M has already contributed to Base SFT, and AC\-JEPA changes how its synthetic pairs are reused after Real\-SFT adaptation\. In a small\-scale implementation, we also used synthetic pairs from MolParser\-7M alongside Real\-SFT for token\-level SFT\. This mixture did not improve over Real\-SFT\-only adaptation and was more prone to overfitting\. In contrast, retaining Real\-SFT token supervision and reusing the synthetic pairs through AC\-JEPA yields the improvements in[Table1](https://arxiv.org/html/2610.36493#S4.T1)\. The pilot therefore supports the importance of the supervision route rather than an explanation based simply on adding paired examples or extending mixed token\-level training\. We report the pilot as qualitative design evidence, with its reporting scope specified in[SectionA\.4](https://arxiv.org/html/2610.36493#A1.SS4.SSS0.Px1)\.

### 4\.3Pseudo\-Label Noise Analysis

##### Model\-generated pseudo\-label noise\.

We first compare two ways of using pseudo\-labels generated for unlabeled open\-environment images: direct token\-level SFT and latent self\-training with AC\-JEPA\. Both settings use model\-generated pseudo\-labels, which contain naturally occurring recognition and SMILES errors rather than manually specified corruptions\.

Table 2:Accuracy \(%\) when model\-generated pseudo\-labels are used for direct token\-level SFT or AC\-JEPA latent self\-training\.Relative to direct pseudo\-label SFT, the first AC\-JEPA self\-training round improves Simple and Markush accuracy by 15\.04 and 6\.74 points, respectively\. The second round increases these margins to 16\.76 and 12\.24 points\. These results show a clear empirical advantage for routing pseudo\-pairs through latent prediction in the current training pipeline\. Because the recorded runs do not yet match total training budget exactly, we treat this comparison as evidence from the realized curriculum rather than as an isolated causal comparison between loss functions\.

##### Controlled synthetic SMILES noise\.

To compare the noise tolerance of direct SFT and AC\-JEPA independently of the unknown error distribution of model\-generated pseudo\-labels, we construct a compact controlled testbed from the clean MolParser Real\-SFT subset\[[8](https://arxiv.org/html/2610.36493#bib.bib7)\]and MolGrapher\-Synthetic\-300K\[[15](https://arxiv.org/html/2610.36493#bib.bib6)\]\. We corrupt only the SMILES labels of a controlled fractionρseq\\rho\_\{\\mathrm\{seq\}\}of the MolGrapher examples; their paired molecular images and all Real\-SFT examples remain unchanged\. For each selected MolGrapher label, the current corruption procedure perturbs 5–15% of its SMILES characters\. In AC\-JEPA training, supervised generation is anchored by the clean Real\-SFT data, whereas the corrupted MolGrapher pairs are used only by the JEPA latent\-prediction objective\. In the direct\-SFT control, both the clean Real\-SFT pairs and the corrupted MolGrapher pairs are optimized with the token\-level SFT objective\. We compare these routes across increasing sequence\-level noise and evaluate both on the unchanged WildMol Simple and Markush benchmarks in[Table3](https://arxiv.org/html/2610.36493#S4.T3)\. Asρseq\\rho\_\{\\mathrm\{seq\}\}rises to 80%, direct SFT degrades sharply, whereas AC\-JEPA retains gains over the clean Base SFT on both benchmarks\.

Table 3:Accuracy \(%\) under controlled SMILES corruption\.ρseq\\rho\_\{\\mathrm\{seq\}\}denotes the fraction of corrupted MolGrapher labels; 5–15% of characters are perturbed in each selected label\.

### 4\.4Ablation Studies

##### Intermediate latent interface\.

The JEPA objective reads an intermediate decoder layer rather than the final generation state\. The current coarse sweep in[Table4](https://arxiv.org/html/2610.36493#S4.T4)favors layer 17 among the four recorded choices, suggesting that states too close to the final decoder output are less effective as predictive targets\.

Table 4:Preliminary Simple accuracy \(%\) for the recorded decoder\-layer choices used as the JEPA latent interface\.
##### Masking configuration\.

[Table5](https://arxiv.org/html/2610.36493#S4.T5)varies the masking ratio and masked modalities under the same structured image\-block and SMILES\-span corruption\. Joint image–SMILES masking at ratio 0\.35 performs best, while lower or higher ratios reduce Simple accuracy\. Both SMILES\-only and image\-only masking remain above Base SFT, indicating that masked prediction in either modality still benefits the SFT task\.

Table 5:Preliminary Simple accuracy \(%\) for the recorded masking\-ratio and masked\-modality variants\.
##### Predictor and latent\-stability diagnostics\.

Additional configuration diagnostics provide empirical support for the predictor and latent\-stability choices used in the final model\. Replacing bidirectional prediction with causal attention substantially slowed predictive MSE convergence and yielded 72\.95% Simple accuracy\. Reducing the JEPA latent dimension from 1024 to 256 similarly slowed convergence and yielded 74\.28% Simple accuracy, suggesting that the JEPA latent should not be aggressively compressed within this decoder architecture\. Removing stop\-gradient caused rapid latent collapse under the default SIGReg setting\. Stronger regularization preserved non\-degenerate latent dispersion, but accuracy degraded rapidly\. At the default weight, pooling image and SMILES latents under one SIGReg term rapidly drove the pooled statistic toward the regularization target, and accuracy degraded substantially\. With a substantially lower pooled\-SIGReg weight, predictive MSE decreased slowly, but Simple accuracy remained limited to 73\.24%\. Together, these diagnostics support bidirectional recovery, clean\-target detachment, and modality\-wise regulation for learning predictable intermediate states without forcing the heterogeneous modalities into one pooled statistic\.

## 5Discussion and Limitations

AC\-JEPA complements rather than replaces SFT: the causal objective anchors the output language and deployable image\-to\-SMILES path, while latent prediction supplies an additional representation\-level signal from gold or pseudo\-labeled pairs\.

Synthetic data serve useful but distinct roles across the curriculum\. Both Base SFT and AC\-JEPA benefit from full MolParser\-7M pretraining before Real\-SFT adaptation; the gold\-pair gain therefore does not come from access to new paired examples\. Our mixed\-SFT pilot indicates that returning synthetic pairs to categorical supervision is not automatically beneficial after real\-image adaptation and can increase overfitting\. Together with the gold\-pair results, this observation supports AC\-JEPA as a way to extract further predictive constraints from existing pairs while retaining a real\-data token anchor\. The pilot remains qualitative, and its scope should be distinguished from a systematic comparison across training budgets and synthetic/real sampling ratios\.

Its noise tolerance is conditional\. Continuous latent targets can soften the effect of local pseudo\-label errors, but systematic bias may still shift the learned representation; our controlled\-corruption results support this behavior only within the tested OCSR setting\. Broader visual\-to\-symbolic generalization therefore requires validation on additional tasks, and the extra pseudo\-label generation and training should not be interpreted as compute\-free\.

## 6Conclusion

We introduced AC\-JEPA, an SFT\-compatible objective that learns visual\-to\-symbolic representations through bidirectional masked latent prediction while retaining causal token supervision\. On OCSR, gold\-pair AC\-JEPA improves a Base SFT model that already shares complete synthetic MolParser\-7M pretraining and Real\-SFT adaptation, showing the usefulness of reusing existing pairs through latent prediction\. The contrasting small\-scale mixed\-SFT observation supports retaining real\-image token supervision while assigning synthetic\-pair reuse to the latent objective\. Latent self\-training further improves performance with imperfect pseudo\-labels, where the recorded direct pseudo\-label SFT run degrades\. These results support anchored joint\-embedding prediction as a practical route to more noise\-tolerant cross\-modal translation\.

## References

- \[1\]M\. Assran, Q\. Duval, I\. Misra, P\. Bojanowski, P\. Vincent, M\. Rabbat, Y\. LeCun, and N\. Ballas\(2023\)Self\-supervised learning from images with a joint\-embedding predictive architecture\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 15619–15629\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52729.2023.01499),[Link](https://openaccess.thecvf.com/content/CVPR2023/html/Assran_Self-Supervised_Learning_From_Images_With_a_Joint-Embedding_Predictive_Architecture_CVPR_2023_paper.html)Cited by:[§1](https://arxiv.org/html/2610.36493#S1.p4.1),[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]M\. Assranet al\.\(2025\)V\-JEPA 2: self\-supervised video models enable understanding, prediction and planning\.arXiv preprint arXiv:2506\.09985\.Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px2.p1.1)\.
- \[3\]A\. Baevski, W\. Hsu, Q\. Xu, A\. Babu, J\. Gu, and M\. Auli\(2022\)Data2vec: a general framework for self\-supervised learning in speech, vision and language\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 1298–1312\.External Links:[Link](https://proceedings.mlr.press/v162/baevski22a.html)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px2.p1.1)\.
- \[4\]S\. Bai, Y\. Cai, R\. Chen,et al\.\(2025\)Qwen3\-VL technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§4\.1](https://arxiv.org/html/2610.36493#S4.SS1.p1.1)\.
- \[5\]R\. Balestriero and Y\. LeCun\(2025\)LeJEPA: provable and scalable self\-supervised learning without the heuristics\.arXiv preprint arXiv:2511\.08544\.Cited by:[§A\.3](https://arxiv.org/html/2610.36493#A1.SS3.p1.2),[§1](https://arxiv.org/html/2610.36493#S1.p6.1),[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.36493#S3.SS2.p2.1)\.
- \[6\]D\. Campos and H\. Ji\(2021\)IMG2SMI: translating molecular structure images to simplified molecular\-input line\-entry system\.arXiv preprint arXiv:2109\.04202\.Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px4.p1.1)\.
- \[7\]D\. Chen, M\. Shukor, T\. Moutakanni, W\. Chung, J\. Yu, T\. Kasarla, A\. Bolourchi, Y\. LeCun, and P\. Fung\(2026\)VL\-JEPA: joint embedding predictive architecture for vision\-language\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tjimrqc2BU)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px2.p1.1)\.
- \[8\]X\. Fang, J\. Wang, X\. Cai, S\. Chen, S\. Yang, H\. Tao, N\. Wang, L\. Yao, L\. Zhang, and G\. Ke\(2025\)MolParser: end\-to\-end visual recognition of molecule structures in the wild\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 24528–24538\.External Links:[Link](https://openaccess.thecvf.com/content/ICCV2025/html/Fang_MolParser_End-to-end_Visual_Recognition_of_Molecule_Structures_in_the_Wild_ICCV_2025_paper.html)Cited by:[§1](https://arxiv.org/html/2610.36493#S1.p1.1),[§1](https://arxiv.org/html/2610.36493#S1.p2.1),[§1](https://arxiv.org/html/2610.36493#S1.p3.1),[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px4.p1.1),[§3\.3](https://arxiv.org/html/2610.36493#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2610.36493#S4.SS1.p1.1),[§4\.3](https://arxiv.org/html/2610.36493#S4.SS3.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.36493#S4.T1)\.
- \[9\]J\. He, J\. Gu, J\. Shen, and M\. Ranzato\(2020\)Revisiting self\-training for neural sequence generation\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SJgdnAVKDH)Cited by:[§1](https://arxiv.org/html/2610.36493#S1.p2.1)\.
- \[10\]K\. He, X\. Chen, S\. Xie, Y\. Li, P\. Dollár, and R\. Girshick\(2022\)Masked autoencoders are scalable vision learners\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16000–16009\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2022/html/He_Masked_Autoencoders_Are_Scalable_Vision_Learners_CVPR_2022_paper.html)Cited by:[§1](https://arxiv.org/html/2610.36493#S1.p4.1),[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px2.p1.1)\.
- \[11\]C\. Jia, Y\. Yang, Y\. Xia, Y\. Chen, Z\. Parekh, H\. Pham, Q\. V\. Le, Y\. Sung, Z\. Li, and T\. Duerig\(2021\)Scaling up visual and vision\-language representation learning with noisy text supervision\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 4904–4916\.External Links:[Link](https://proceedings.mlr.press/v139/jia21b.html)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]D\. Lee\(2013\)Pseudo\-label: the simple and efficient semi\-supervised learning method for deep neural networks\.InICML Workshop on Challenges in Representation Learning,Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px3.p1.1)\.
- \[13\]J\. Li, D\. Li, C\. Xiong, and S\. Hoi\(2022\)BLIP: bootstrapping language\-image pre\-training for unified vision\-language understanding and generation\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 12888–12900\.External Links:[Link](https://proceedings.mlr.press/v162/li22n.html)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]J\. Li, R\. R\. Selvaraju, A\. D\. Gotmare, S\. Joty, C\. Xiong, and S\. C\. H\. Hoi\(2021\)Align before fuse: vision and language representation learning with momentum distillation\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 9694–9705\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/hash/505259756244493872b7709a8a01b536-Abstract.html)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px3.p1.1)\.
- \[15\]L\. Morin, M\. Danelljan, M\. I\. Agea, A\. Nassar, V\. Weber, I\. Meijer, P\. Staar, and F\. Yu\(2023\)MolGrapher: graph\-based visual recognition of chemical structures\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 19552–19561\.External Links:[Link](https://openaccess.thecvf.com/content/ICCV2023/html/Morin_MolGrapher_Graph-based_Visual_Recognition_of_Chemical_Structures_ICCV_2023_paper.html)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px4.p1.1),[§4\.3](https://arxiv.org/html/2610.36493#S4.SS3.SSS0.Px2.p1.1)\.
- \[16\]A\. Perera, K\. Hewagamage, S\. Nazar, K\. Abeywardana, H\. Gallella, R\. Rodrigo, and M\. Afham\(2025\)CrossJEPA: cross\-modal joint\-embedding predictive architecture for efficient 3d representation learning from 2d images\.arXiv preprint arXiv:2511\.18424\.Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px2.p1.1)\.
- \[17\]Y\. Qian, J\. Guo, Z\. Tu, Z\. Li, C\. W\. Coley, and R\. Barzilay\(2023\)MolScribe: robust molecular structure recognition with image\-to\-graph generation\.Journal of Chemical Information and Modeling63\(7\),pp\. 1925–1934\.External Links:[Document](https://dx.doi.org/10.1021/acs.jcim.2c01480),[Link](https://pubs.acs.org/doi/10.1021/acs.jcim.2c01480)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px4.p1.1)\.
- \[18\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever\(2021\)Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.External Links:[Link](https://proceedings.mlr.press/v139/radford21a.html)Cited by:[§1](https://arxiv.org/html/2610.36493#S1.p4.1),[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px1.p1.1)\.
- \[19\]K\. Sohn, D\. Berthelot, N\. Carlini, Z\. Zhang, H\. Zhang, C\. A\. Raffel, E\. D\. Cubuk, A\. Kurakin, and C\. Li\(2020\)FixMatch: simplifying semi\-supervised learning with consistency and confidence\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 596–608\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/06964dce9addb1c5cb5d6e3d9838f733-Abstract.html)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px3.p1.1)\.
- \[20\]H\. Song, M\. Kim, D\. Park, Y\. Shin, and J\. Lee\(2023\)Learning from noisy labels with deep neural networks: a survey\.IEEE Transactions on Neural Networks and Learning Systems34\(11\),pp\. 8135–8153\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2022.3152527),[Link](https://doi.org/10.1109/TNNLS.2022.3152527)Cited by:[§1](https://arxiv.org/html/2610.36493#S1.p2.1),[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px3.p1.1)\.
- \[21\]R\. Teotia, A\. Bardes, M\. Rabbat, S\. Chopra, M\. J\. Muckley, and N\. Ballas\(2026\)MJEPA: a simple and scalable joint\-embedding predictive architecture for audio\-visual learning\.arXiv preprint arXiv:2606\.25225\.Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px2.p1.1)\.
- \[22\]W\. Wang, H\. Bao, L\. Dong, J\. Bjorck, Z\. Peng, Q\. Liu, K\. Aggarwal, O\. K\. Mohammed, S\. Singhal, S\. Som, and F\. Wei\(2023\)Image as a foreign language: BEiT pretraining for vision and vision\-language tasks\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19175–19186\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2023/html/Wang_Image_as_a_Foreign_Language_BEiT_Pretraining_for_Vision_and_CVPR_2023_paper.html)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px1.p1.1)\.
- \[23\]D\. Weininger\(1988\)SMILES, a chemical language and information system\. 1\. introduction to methodology and encoding rules\.Journal of Chemical Information and Computer Sciences28\(1\),pp\. 31–36\.External Links:[Document](https://dx.doi.org/10.1021/ci00057a005),[Link](https://pubs.acs.org/doi/10.1021/ci00057a005)Cited by:[§1](https://arxiv.org/html/2610.36493#S1.p1.1),[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px4.p1.1)\.
- \[24\]Q\. Xie, M\. Luong, E\. Hovy, and Q\. V\. Le\(2020\)Self\-training with noisy student improves imagenet classification\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10687–10698\.External Links:[Link](https://openaccess.thecvf.com/content_CVPR_2020/html/Xie_Self-Training_With_Noisy_Student_Improves_ImageNet_Classification_CVPR_2020_paper.html)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px3.p1.1)\.
- \[25\]H\. Yang, H\. Wang, C\. Zhu, J\. Wang, L\. Li, H\. Lai, H\. Ao, Y\. Lyu, J\. Wu, J\. Sun, L\. Chen, Y\. Cao, R\. Zhang, S\. Lu, L\. Wu, B\. Wang, and C\. He\(2026\)MolRecBench\-Wild: a real\-world benchmark for optical chemical structure recognition\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,External Links:[Link](https://arxiv.org/abs/2605.05832)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px4.p1.1)\.
- \[26\]W\. Zhang, X\. Wang, B\. Feng, and W\. Liu\(2026\)MolSight: optical chemical structure recognition with SMILES pretraining, multi\-granularity learning and reinforcement learning\.Proceedings of the AAAI Conference on Artificial Intelligence40\(15\),pp\. 12825–12833\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i15.38280),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/38280)Cited by:[§2](https://arxiv.org/html/2610.36493#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix AAppendix

### A\.1Structured Corruption Details

The main text defines the image and SMILES mask sets used by the latent MSE\. For completeness, contiguous spans are sampled only within the assistant SMILES region,

ℳsmi\\displaystyle\\mathcal\{M\}\_\{\\mathrm\{smi\}\}=⋃b=1Bsmi\{sb,…,sb\+ℓb−1\},\\displaystyle=\\bigcup\_\{b=1\}^\{B\_\{\\mathrm\{smi\}\}\}\\\{s\_\{b\},\\ldots,s\_\{b\}\+\\ell\_\{b\}\-1\\\},\(13\)ℓb\\displaystyle\\ell\_\{b\}∼Unif⁡\[ℓmin,ℓmax\],\\displaystyle\\sim\\mathrm\{Unif\}\[\\ell\_\{\\min\},\\ell\_\{\\max\}\],until the prescribed SMILES masking ratio is reached\. Rectangular blocks are sampled on the visual patch grid,

ℳimg=⋃b=1Bimg\{\(u,v\):ub≤u<ub\+hb,vb≤v<vb\+wb\}\.\\mathcal\{M\}\_\{\\mathrm\{img\}\}=\\bigcup\_\{b=1\}^\{B\_\{\\mathrm\{img\}\}\}\\left\\\{\(u,v\):\\begin\{aligned\} u\_\{b\}&\\leq u<u\_\{b\}\+h\_\{b\},\\\\\[\-2\.0pt\] v\_\{b\}&\\leq v<v\_\{b\}\+w\_\{b\}\\end\{aligned\}\\right\\\}\.\(14\)These masks can remove all or part of local symbolic motifs and coherent image regions without asserting that every sampled mask coincides with a complete chemical substructure\. Prompt, padding, and special tokens are excluded from both sets\. The masks are sampled before the clean and corrupted forward passes so that both branches use the same prediction positions\. If a modality has no masked positions in a micro\-batch, its MSE term is skipped through a graph\-valid zero\.

### A\.2Target Stabilization and Gradient Routing

The clean branch produces the undetached latentZcleanZ\_\{\\mathrm\{clean\}\}, from which the predictive target is formed asZtgt=sg⁡\(Zclean\)Z\_\{\\mathrm\{tgt\}\}=\\operatorname\{sg\}\(Z\_\{\\mathrm\{clean\}\}\)\. Consequently,ℒMSE\\mathcal\{L\}\_\{\\mathrm\{MSE\}\}updates the corrupted\-view backbone up to layerℓ\\ell, the shared projector, and the predictor, but does not move the clean target through the MSE path\. The stop\-gradient does not freeze the shared backbone\. The gold\-pair SFT loss still updates the complete visual encoder and causal decoder, and modality\-wise SIGReg is evaluated on the undetachedZcleanZ\_\{\\mathrm\{clean\}\}\. Thus, stop\-gradient stabilizes the clean SFT\-anchored target, whereas SIGReg supplies collapse control\.

The no\-detach variant allows the MSE to update both clean and corrupted representations and is retained as a target\-stability diagnostic\. It is not treated as the default merely because the underlying LeJEPA regularizer can be used without a detached target\. This distinction is specific to the present shared\-weight autoregressive generator, for which a latent space that is easy to predict need not remain useful for exact SMILES decoding\.

### A\.3Modality\-wise SIGReg Details

SIGReg compares one\-dimensional random projections of the clean latent distribution with the characteristic function of a standard normal target\[[5](https://arxiv.org/html/2610.36493#bib.bib12)\]\. For modalitymm, let𝒵m\\mathcal\{Z\}\_\{m\}contain all eligible clean\-view latents in the JEPA mini\-batch and letNm=\|𝒵m\|N\_\{m\}=\|\\mathcal\{Z\}\_\{m\}\|\. With random unit directions

a~pm∼𝒩⁡\(0,I\),apm=a~pm/‖a~pm‖2,\\widetilde\{a\}\_\{p\}^\{m\}\\sim\\mathcal\{N\}\(0,I\),\\qquad a\_\{p\}^\{m\}=\\widetilde\{a\}\_\{p\}^\{m\}/\\\|\\widetilde\{a\}\_\{p\}^\{m\}\\\|\_\{2\},\(15\)and frequency knotstkt\_\{k\}, the modality\-specific penalty is

ℒSIGRegm\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}^\{m\}=1P​∑p=1PNm​∑k=1Kwk\\displaystyle=\\frac\{1\}\{P\}\\sum\_\{p=1\}^\{P\}N\_\{m\}\\sum\_\{k=1\}^\{K\}w\_\{k\}\(16\)×\|1Nm∑z∈𝒵mei​tk​\(apm\)⊤​z\\displaystyle\\times\\left\|\\frac\{1\}\{N\_\{m\}\}\\sum\_\{z\\in\\mathcal\{Z\}\_\{m\}\}e^\{\\mathrm\{i\}t\_\{k\}\(a\_\{p\}^\{m\}\)^\{\\top\}z\}\\right\.−e−tk2/2\|2\.\\displaystyle\\left\.\\qquad\-e^\{\-t\_\{k\}^\{2\}/2\}\\right\|^\{2\}\.Image and SMILES positions are pooled across examples within their respective modality, but the two modalities are never pooled into one SIGReg statistic\. Because the statistic is computed from the clean branch, active mask tokens do not enter𝒵m\\mathcal\{Z\}\_\{m\}\. The main\-text regularizer in[Equation5](https://arxiv.org/html/2610.36493#S3.E5)is the weighted sum of these two penalties\.

### A\.4Training\-Stream Details

The unlabeled pool used for R1 and R2 includes molecular images from USPTO\-Mol \([https://huggingface\.co/datasets/xiegeo/uspto\-mol](https://huggingface.co/datasets/xiegeo/uspto-mol)\) and structure figures extracted from scientific papers\. The coefficientsλsft\\lambda\_\{\\mathrm\{sft\}\},λmsem\\lambda\_\{\\mathrm\{mse\}\}^\{m\}, andλsigm\\lambda\_\{\\mathrm\{sig\}\}^\{m\}are fixed within a training run\. SFT and JEPA are estimated from separate mini\-batches drawn from𝒟sft\\mathcal\{D\}\_\{\\mathrm\{sft\}\}and𝒟jepag\\mathcal\{D\}\_\{\\mathrm\{jepa\}\}^\{g\}, respectively, and may use different batch sizes; this changes the stochastic estimator but not the objectives in[Equations6](https://arxiv.org/html/2610.36493#S3.E6)and[7](https://arxiv.org/html/2610.36493#S3.E7)\. For the simplest accepted pseudo\-label pool,q⁡\(I,S^\)=1q\(I,\\hat\{S\}\)=1\. Algorithm[1](https://arxiv.org/html/2610.36493#alg1)makes the reported refresh protocol explicit\. The common SFT\-anchored, JEPA\-warmed checkpointC0C\_\{0\}generates the first pseudo\-pair pool and initializes the round\-1 student\. The trained round\-1 checkpoint generates the second pool, but the round\-2 student is reinitialized fromC0C\_\{0\}rather than continued from the round\-1 weights\.

##### Synthetic\-pair reuse pilot\.

Our small\-scale mixed\-SFT implementation sampled synthetic pairs from MolParser\-7M together with Real\-SFT for categorical token supervision\. Relative to Real\-SFT\-only adaptation, it provided no improvement and was more prone to overfitting\. This is a qualitative observation motivating the choice to retain real\-image token supervision and reuse synthetic pairs through latent prediction\. It is separate from the controlled label\-corruption experiment, which uses MolGrapher\-Synthetic\-300K\. We do not assign a numerical endpoint or claim a complete 7M\-example rerun or a matched\-compute comparison for this pilot\.

Algorithm 1AC\-JEPA training and pseudo\-label refreshInput: common checkpointC0C\_\{0\}; reliable SFT pairs𝒟sft\\mathcal\{D\}\_\{\\mathrm\{sft\}\}; gold JEPA pairs𝒟jepag\\mathcal\{D\}\_\{\\mathrm\{jepa\}\}^\{g\}; unlabeled images𝒟u\\mathcal\{D\}\_\{u\} Output: independently trained checkpointsCg,C1,C2C\_\{g\},C\_\{1\},C\_\{2\} Notation:ℓMSEm\\ell\_\{\\mathrm\{MSE\}\}^\{m\}andℓSIGRegm\\ell\_\{\\mathrm\{SIGReg\}\}^\{m\}are unweighted modality\-specific terms

1:procedure

Train​\(C0,𝒟p,η,T\)\\textsc\{Train\}\(C\_\{0\},\\mathcal\{D\}\_\{p\},\\eta,T\)
2:Restore

C0C\_\{0\}and initialize a run\-specific optimizer

3:for

t=1,…,Tt=1,\\ldots,Tdo

4:Sample

Bs∼𝒟sftB\_\{s\}\\\!\\sim\\\!\\mathcal\{D\}\_\{\\mathrm\{sft\}\}and

Bg∼𝒟jepagB\_\{g\}\\\!\\sim\\\!\\mathcal\{D\}\_\{\\mathrm\{jepa\}\}^\{g\}
5:

ℒ←λsft​ℒSFT​\(Bs\)\\mathcal\{L\}\\leftarrow\\lambda\_\{\\mathrm\{sft\}\}\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(B\_\{s\}\)
6:

𝒬←\{\(Bg,1\)\}\\mathcal\{Q\}\\leftarrow\\\{\(B\_\{g\},1\)\\\}
7:if

𝒟p≠∅\\mathcal\{D\}\_\{p\}\\neq\\emptysetthen

8:Sample

Bp∼𝒟pB\_\{p\}\\sim\\mathcal\{D\}\_\{p\}and add

\(Bp,η​q​\(Bp\)\)\(B\_\{p\},\\eta q\(B\_\{p\}\)\)to

𝒬\\mathcal\{Q\}
9:endif

10:foreach

\(B,w\)∈𝒬\(B,w\)\\in\\mathcal\{Q\}do

11:Construct clean and structured\-masked views of

BB
12:

Zc←gϕ​\(Hθℓ​\(𝐕clean\)\)Z\_\{c\}\\leftarrow g\_\{\\phi\}\(H\_\{\\theta\}^\{\\ell\}\(\\mathbf\{V\}\_\{\\mathrm\{clean\}\}\)\)
13:

Zt←sg⁡\(Zc\)Z\_\{t\}\\leftarrow\\operatorname\{sg\}\(Z\_\{c\}\)
14:

Zp←pψ​\(gϕ​\(Hθℓ​\(𝐕mask\)\)\)Z\_\{p\}\\leftarrow p\_\{\\psi\}\(g\_\{\\phi\}\(H\_\{\\theta\}^\{\\ell\}\(\\mathbf\{V\}\_\{\\mathrm\{mask\}\}\)\)\)
15:

ℒpred←∑mλmsem​ℓMSEm​\(Zp,Zt\)\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\\leftarrow\\sum\_\{m\}\\lambda\_\{\\mathrm\{mse\}\}^\{m\}\\ell\_\{\\mathrm\{MSE\}\}^\{m\}\(Z\_\{p\},Z\_\{t\}\)
16:

ℒreg←∑mλsigm​ℓSIGRegm​\(Zc\)\\mathcal\{L\}\_\{\\mathrm\{reg\}\}\\leftarrow\\sum\_\{m\}\\lambda\_\{\\mathrm\{sig\}\}^\{m\}\\ell\_\{\\mathrm\{SIGReg\}\}^\{m\}\(Z\_\{c\}\)
17:

ℒ←ℒ\+w⁡\(ℒpred\+ℒreg\)\\mathcal\{L\}\\leftarrow\\mathcal\{L\}\+w\(\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\+\\mathcal\{L\}\_\{\\mathrm\{reg\}\}\)
18:endfor

19:Update

\(θ,ϕ,ψ\)\(\\theta,\\phi,\\psi\)using

∇ℒ\\nabla\\mathcal\{L\}
20:endfor

21:returnthe trained checkpoint

22:end procedure

23:

Cg←Train​\(C0,∅,0,Tg\)C\_\{g\}\\leftarrow\\textsc\{Train\}\(C\_\{0\},\\emptyset,0,T\_\{g\}\)
24:

𝒟p\(1\)←\{\(I,Decode⁡\(C0,I\)\):I∈𝒟u\}\\mathcal\{D\}\_\{p\}^\{\(1\)\}\\leftarrow\\\{\(I,\\operatorname\{Decode\}\(C\_\{0\},I\)\):I\\in\\mathcal\{D\}\_\{u\}\\\}
25:

C1←Train​\(C0,𝒟p\(1\),η1,T1\)C\_\{1\}\\leftarrow\\textsc\{Train\}\(C\_\{0\},\\mathcal\{D\}\_\{p\}^\{\(1\)\},\\eta\_\{1\},T\_\{1\}\)
26:

𝒟p\(2\)←\{\(I,Decode⁡\(C1,I\)\):I∈𝒟u\}\\mathcal\{D\}\_\{p\}^\{\(2\)\}\\leftarrow\\\{\(I,\\operatorname\{Decode\}\(C\_\{1\},I\)\):I\\in\\mathcal\{D\}\_\{u\}\\\}
27:

C2←Train​\(C0,𝒟p\(2\),η2,T2\)C\_\{2\}\\leftarrow\\textsc\{Train\}\(C\_\{0\},\\mathcal\{D\}\_\{p\}^\{\(2\)\},\\eta\_\{2\},T\_\{2\}\)
28:return

Cg,C1,C2C\_\{g\},C\_\{1\},C\_\{2\}

### A\.5Supporting Analysis and Proofs for Task Readout Stability and Pseudo\-Label Noise

This appendix proves the conditional\-readout and parameter\-influence results in the main text and makes their assumptions explicit\. The first analysis holds a jointly trained model fixed, whereas the second is local to a counterfactual clean\-pair solution anchored by the gold SFT and JEPA objectives\. It does not assert global robustness or compare loss functions independently of representation geometry\.

#### A\.5\.1From Symbolic JEPA Loss to Conditional Residual

Sampleiifrom the masked SMILES positions under the training data and masking distribution\. WriteZi=Ztgt,iZ\_\{i\}=Z\_\{\\mathrm\{tgt\},i\},Z^i=Zpred,i\\widehat\{Z\}\_\{i\}=Z\_\{\\mathrm\{pred\},i\}, and letCiC\_\{i\}contain the corrupted multimodal latent sequence and mask\-position information available to the predictor\. The conditional meanmi​\(Ci\)=𝔼⁡\[Zi∣Ci\]m\_\{i\}\(C\_\{i\}\)=\\mathbb\{E\}\[Z\_\{i\}\\mid C\_\{i\}\]gives the exact orthogonal decomposition of the unweighted symbolic MSE component,

ℒMSEsmi\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{MSE\}\}^\{\\mathrm\{smi\}\}=𝔼​‖Zi−Z^i‖22\\displaystyle=\\mathbb\{E\}\\\|Z\_\{i\}\-\\widehat\{Z\}\_\{i\}\\\|\_\{2\}^\{2\}\(17\)=𝔼​‖Zi−mi​\(Ci\)‖22⏟R¯Jsmi\+𝔼​‖mi​\(Ci\)−Z^i‖22⏟ℰpredsmi\.\\displaystyle=\\underbrace\{\\mathbb\{E\}\\\|Z\_\{i\}\-m\_\{i\}\(C\_\{i\}\)\\\|\_\{2\}^\{2\}\}\_\{\\overline\{R\}\_\{\\mathrm\{J\}\}^\{\\mathrm\{smi\}\}\}\+\\underbrace\{\\mathbb\{E\}\\\|m\_\{i\}\(C\_\{i\}\)\-\\widehat\{Z\}\_\{i\}\\\|\_\{2\}^\{2\}\}\_\{\\mathcal\{E\}\_\{\\mathrm\{pred\}\}^\{\\mathrm\{smi\}\}\}\.The cross term vanishes because𝔼⁡\[Zi−mi​\(Ci\)∣Ci\]=0\\mathbb\{E\}\[Z\_\{i\}\-m\_\{i\}\(C\_\{i\}\)\\mid C\_\{i\}\]=0\. ThusR¯Jsmi\\overline\{R\}\_\{\\mathrm\{J\}\}^\{\\mathrm\{smi\}\}is the Bayes residual of symbolic latent prediction andℰpredsmi\\mathcal\{E\}\_\{\\mathrm\{pred\}\}^\{\\mathrm\{smi\}\}measures predictor approximation and optimization error\. Stop\-gradient changes the update path throughZiZ\_\{i\}but not this forward\-risk identity\.

Consider two checkpoints evaluated under the same data and masking distribution\. LetΔJ=ℒMSE,0smi−ℒMSE,1smi\\Delta\_\{\\mathrm\{J\}\}=\\mathcal\{L\}\_\{\\mathrm\{MSE\},0\}^\{\\mathrm\{smi\}\}\-\\mathcal\{L\}\_\{\\mathrm\{MSE\},1\}^\{\\mathrm\{smi\}\}, and assume that predictor warm\-up givesℰpred,0smi≤ϵp\\mathcal\{E\}\_\{\\mathrm\{pred\},0\}^\{\\mathrm\{smi\}\}\\leq\\epsilon\_\{\\mathrm\{p\}\}\. Since predictor excess risk is nonnegative,[Equation17](https://arxiv.org/html/2610.36493#A1.E17)yields

R¯J,0smi−R¯J,1smi=ΔJ−ℰpred,0smi\+ℰpred,1smi≥ΔJ−ϵp\.\\overline\{R\}\_\{\\mathrm\{J\},0\}^\{\\mathrm\{smi\}\}\-\\overline\{R\}\_\{\\mathrm\{J\},1\}^\{\\mathrm\{smi\}\}=\\Delta\_\{\\mathrm\{J\}\}\-\\mathcal\{E\}\_\{\\mathrm\{pred\},0\}^\{\\mathrm\{smi\}\}\+\\mathcal\{E\}\_\{\\mathrm\{pred\},1\}^\{\\mathrm\{smi\}\}\\geq\\Delta\_\{\\mathrm\{J\}\}\-\\epsilon\_\{\\mathrm\{p\}\}\.\(18\)Hence a symbolic\-MSE decrease larger than the initial predictor excess reduces the Bayes latent residual\. The statement concerns the population objective at the two checkpoints; it does not require every stochastic update to decrease the residual\.

To connect this average residual to[Theorem3\.1](https://arxiv.org/html/2610.36493#S3.Thmproposition1), definemi,s\+\(Ci\)=𝔼\[Zi∣Ci,Si\+1=s\+\]m\_\{i,s^\{\+\}\}\(C\_\{i\}\)=\\mathbb\{E\}\[Z\_\{i\}\\mid C\_\{i\},S\_\{i\+1\}=s^\{\+\}\]\. The law of total variance gives

R¯Jsmi\\displaystyle\\overline\{R\}\_\{\\mathrm\{J\}\}^\{\\mathrm\{smi\}\}=𝔼​‖Zi−mi,Si\+1​\(Ci\)‖22\\displaystyle=\\mathbb\{E\}\\\|Z\_\{i\}\-m\_\{i,S\_\{i\+1\}\}\(C\_\{i\}\)\\\|\_\{2\}^\{2\}\(19\)\+𝔼​‖mi,Si\+1​\(Ci\)−mi​\(Ci\)‖22\.\\displaystyle\+\\mathbb\{E\}\\\|m\_\{i,S\_\{i\+1\}\}\(C\_\{i\}\)\-m\_\{i\}\(C\_\{i\}\)\\\|\_\{2\}^\{2\}\.Under the locally affine latent interfaceZi=Ji​Hi\+aiZ\_\{i\}=J\_\{i\}H\_\{i\}\+a\_\{i\}, the first term equals𝔼⁡\[RJ,i​\(Ci\)\]\\mathbb\{E\}\[R\_\{\\mathrm\{J\},i\}\(C\_\{i\}\)\]; for a nonlinear interface this equality holds to first order with the corresponding Taylor remainder\. Suppose the overlap constantκ\\kappais uniform over the analyzed strata and the SFT anchor maintainsμi,s−​\(Ci\)≥μ0\>0\\mu\_\{i,s^\{\-\}\}\(C\_\{i\}\)\\geq\\mu\_\{0\}\>0for a fixed competitor assignment\. Applying[Theorem3\.1](https://arxiv.org/html/2610.36493#S3.Thmproposition1), averaging over contexts and next\-token classes, and using concavity ofx/\(x\+μ02\)x/\(x\+\\mu\_\{0\}^\{2\}\)gives

Pr\[Δi,s−\(Hi\)≤0\]≤κ​R¯Jsmiκ​R¯Jsmi\+μ02\.\\Pr\[\\Delta\_\{i,s^\{\-\}\}\(H\_\{i\}\)\\leq 0\]\\leq\\frac\{\\kappa\\,\\overline\{R\}\_\{\\mathrm\{J\}\}^\{\\mathrm\{smi\}\}\}\{\\kappa\\,\\overline\{R\}\_\{\\mathrm\{J\}\}^\{\\mathrm\{smi\}\}\+\\mu\_\{0\}^\{2\}\}\.\(20\)Together,[Equations18](https://arxiv.org/html/2610.36493#A1.E18)and[20](https://arxiv.org/html/2610.36493#A1.E20)connect a decrease in the actual symbolic JEPA objective to a tighter averaged readout bound under predictor tracking, local decoder–JEPA overlap, and SFT margin preservation\. Modality\-wise SIGReg excludes the trivial collapsed route to a small latent residual but does not replace these conditions\. Under these conditions, the SFT and JEPA objectives play complementary roles\. SFT maintains the class\-conditional mean margin, while a reduction in symbolic JEPA risk reduces context\-unpredictable latent variation\. Decoder–JEPA overlap converts this reduction into a tighter bound on the probability that hidden\-state fluctuations cross the decision boundary\.

##### Image\-to\-symbol transfer\.

The preceding argument uses symbolic MSE because the corresponding latent state directly supports the causal decoder, but image\-side prediction can also reduce symbolic decision variation through cross\-modal fusion\. LetVVandSSdenote image\- and symbol\-side intermediate states and locally write the fused symbolic state asHi=Fi​\(V,S\)H\_\{i\}=F\_\{i\}\(V,S\)\. Around the conditional mean, its centered residual admits the first\-order decomposition

Ui=Ai​Uimg\+Bi​Usmi\+ri,U\_\{i\}=A\_\{i\}U\_\{\\mathrm\{img\}\}\+B\_\{i\}U\_\{\\mathrm\{smi\}\}\+r\_\{i\},\(21\)whereAiA\_\{i\}is the image\-to\-symbol Jacobian induced by cross\-modal interaction,BiB\_\{i\}is the symbol\-side Jacobian, andrir\_\{i\}is the linearization remainder\. For modality\-specific latent interfacesJimgJ\_\{\\mathrm\{img\}\}andJsmiJ\_\{\\mathrm\{smi\}\}, defineRJm=𝔼\[∥JmUm∥22∣Ci,Si\+1=s\+\]R\_\{\\mathrm\{J\}\}^\{m\}=\\mathbb\{E\}\[\\\|J\_\{m\}U\_\{m\}\\\|\_\{2\}^\{2\}\\mid C\_\{i\},S\_\{i\+1\}=s^\{\+\}\]\. IfAi⊤​di,s−​di,s−⊤​Ai⪯κimg​Jimg⊤​JimgA\_\{i\}^\{\\top\}d\_\{i,s^\{\-\}\}d\_\{i,s^\{\-\}\}^\{\\top\}A\_\{i\}\\preceq\\kappa\_\{\\mathrm\{img\}\}J\_\{\\mathrm\{img\}\}^\{\\top\}J\_\{\\mathrm\{img\}\}and the analogous symbol\-side overlap holds withκsmi\\kappa\_\{\\mathrm\{smi\}\}, then

vi,s−​\(Ci\)≤κimg​RJimg\+κsmi​RJsmi\+ϵcross\+ϵlin,v\_\{i,s^\{\-\}\}\(C\_\{i\}\)\\leq\\kappa\_\{\\mathrm\{img\}\}R\_\{\\mathrm\{J\}\}^\{\\mathrm\{img\}\}\+\\kappa\_\{\\mathrm\{smi\}\}R\_\{\\mathrm\{J\}\}^\{\\mathrm\{smi\}\}\+\\epsilon\_\{\\mathrm\{cross\}\}\+\\epsilon\_\{\\mathrm\{lin\}\},\(22\)whereϵcross\\epsilon\_\{\\mathrm\{cross\}\}bounds cross\-modal covariance terms andϵlin\\epsilon\_\{\\mathrm\{lin\}\}bounds the contribution ofrir\_\{i\}\. Applying the decomposition in[Equation17](https://arxiv.org/html/2610.36493#A1.E17)to masked image positions gives the analogous image\-side loss–residual link; connecting its population average toRJimgR\_\{\\mathrm\{J\}\}^\{\\mathrm\{img\}\}additionally requires the paired sampling and local transfer assumptions above\. Thus image\-side stabilization can tighten the decision\-reversal bound in[Theorem3\.1](https://arxiv.org/html/2610.36493#S3.Thmproposition1)when cross\-modal transfer reaches the decoder\-relevant subspace, the remaining terms are controlled, and SFT preserves the margin\. With locally fixed attention scores,AiA\_\{i\}reduces to attention\-weighted value transfer; score variation is absorbed into the remainder\. The fact that both modalities describe the same entity makes the required alignment plausible but does not guarantee it: no improvement follows if the decoder ignores the image subspace or if correlated or systematic errors dominate\. BecauseCiC\_\{i\}andHiH\_\{i\}in[Theorem3\.1](https://arxiv.org/html/2610.36493#S3.Thmproposition1)are already multimodal, this decomposition makes an implicit pathway explicit rather than introducing a separate guarantee\.

###### Proof of[Theorem3\.1](https://arxiv.org/html/2610.36493#S3.Thmproposition1)\.

Fixi∈ℳsmii\\in\\mathcal\{M\}\_\{\\mathrm\{smi\}\}and condition on\(Ci,Si\+1=s\+\)\(C\_\{i\},S\_\{i\+1\}=s^\{\+\}\)\. The stateHiH\_\{i\}is aligned withSiS\_\{i\}, whereas the causal readout at the same decoder position predictsSi\+1S\_\{i\+1\}\. With the other causal states fixed, writeUi=Hi−𝔼\[Hi∣Ci,Si\+1=s\+\]U\_\{i\}=H\_\{i\}\-\\mathbb\{E\}\[H\_\{i\}\\mid C\_\{i\},S\_\{i\+1\}=s^\{\+\}\]\. Under the locally affine readout,

Δi,s−​\(Hi\)−μi,s−​\(Ci\)\\displaystyle\\Delta\_\{i,s^\{\-\}\}\(H\_\{i\}\)\-\\mu\_\{i,s^\{\-\}\}\(C\_\{i\}\)=di,s−⊤​Ui,\\displaystyle=d\_\{i,s^\{\-\}\}^\{\\top\}U\_\{i\},\(23\)vi,s−​\(Ci\)\\displaystyle v\_\{i,s^\{\-\}\}\(C\_\{i\}\)=𝔼⁡\[\(di,s−⊤​Ui\)2\|Ci,Si\+1=s\+\]\.\\displaystyle=\\mathbb\{E\}\\\!\\left\[\(d\_\{i,s^\{\-\}\}^\{\\top\}U\_\{i\}\)^\{2\}\\,\\middle\|\\,\\begin\{subarray\}\{c\}C\_\{i\},\\\\ S\_\{i\+1\}=s^\{\+\}\\end\{subarray\}\\right\]\.Cantelli’s inequality gives

Pr⁡\[Δi,s−​\(Hi\)≤0\|Ci,Si\+1=s\+\]≤vi,s−​\(Ci\)vi,s−​\(Ci\)\+μi,s−​\(Ci\)2\.\\Pr\\\!\\left\[\\Delta\_\{i,s^\{\-\}\}\(H\_\{i\}\)\\leq 0\\,\\middle\|\\,\\begin\{subarray\}\{c\}C\_\{i\},\\\\ S\_\{i\+1\}=s^\{\+\}\\end\{subarray\}\\right\]\\leq\\frac\{v\_\{i,s^\{\-\}\}\(C\_\{i\}\)\}\{v\_\{i,s^\{\-\}\}\(C\_\{i\}\)\+\\mu\_\{i,s^\{\-\}\}\(C\_\{i\}\)^\{2\}\}\.\(24\)Moreover, the decoder–JEPA overlap conditiondi,s−​di,s−⊤⪯κ​Ji⊤​Jid\_\{i,s^\{\-\}\}d\_\{i,s^\{\-\}\}^\{\\top\}\\preceq\\kappa J\_\{i\}^\{\\top\}J\_\{i\}implies

vi,s−​\(Ci\)\\displaystyle v\_\{i,s^\{\-\}\}\(C\_\{i\}\)=𝔼\[Ui⊤di,s−di,s−⊤Ui∣Ci,Si\+1=s\+\]\\displaystyle=\\mathbb\{E\}\[U\_\{i\}^\{\\top\}d\_\{i,s^\{\-\}\}d\_\{i,s^\{\-\}\}^\{\\top\}U\_\{i\}\\mid C\_\{i\},S\_\{i\+1\}=s^\{\+\}\]\(25\)≤κ𝔼\[∥JiUi∥22∣Ci,Si\+1=s\+\]\.\\displaystyle\\leq\\kappa\\mathbb\{E\}\[\\\|J\_\{i\}U\_\{i\}\\\|\_\{2\}^\{2\}\\mid C\_\{i\},S\_\{i\+1\}=s^\{\+\}\]\.\(26\)The functionv/\(v\+μ2\)v/\(v\+\\mu^\{2\}\)is increasing invvand decreasing in positiveμ\\mu, which proves the final comparison\. For a nonlinear JEPA interface,Ji​UiJ\_\{i\}U\_\{i\}is the first\-order projected residual; the theorem therefore applies only where the local linearization and overlap condition are valid\. ∎

#### A\.5\.2Pseudo\-label Residuals and Parameter Influence

The second result compares two training problems that use the same images and differ only in the symbolic sequence attached to each pseudo\-pair\. Let

ℒclean​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{clean\}\}\(\\theta\)=ℒanchor​\(θ\)\+λn​∑j=1nℓj​\(θ,Sj\),\\displaystyle=\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\(\\theta\)\+\\frac\{\\lambda\}\{n\}\\sum\_\{j=1\}^\{n\}\\ell\_\{j\}\(\\theta;S\_\{j\}\),\(27\)ℒpseudo​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{pseudo\}\}\(\\theta\)=ℒanchor​\(θ\)\+λn​∑j=1nℓj​\(θ,S^j\),\\displaystyle=\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\(\\theta\)\+\\frac\{\\lambda\}\{n\}\\sum\_\{j=1\}^\{n\}\\ell\_\{j\}\(\\theta;\\widehat\{S\}\_\{j\}\),ℓj​\(θ,S\)\\displaystyle\\ell\_\{j\}\(\\theta;S\)=12​‖rj​\(θ,S\)‖22\.\\displaystyle=\\frac\{1\}\{2\}\\\|r\_\{j\}\(\\theta;S\)\\\|\_\{2\}^\{2\}\.ℒanchor\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}contains the gold\-pair SFT and JEPA streams, andθ⋆\\theta^\{\\star\}is a local stationary point ofℒclean\\mathcal\{L\}\_\{\\mathrm\{clean\}\}\. The unobserved sequencesSjS\_\{j\}define a counterfactual reference and are not required by the training algorithm\. Using the same images in both objectives separates pseudo\-label corruption from the useful signal that can arise simply from adding open\-environment diagrams\. The factor1/21/2only absorbs the conventional factor of two intoλ\\lambda\.

For a fixed image, mask realization, and predicted latent position, write

rjc\\displaystyle r\_\{j\}^\{\\mathrm\{c\}\}=Zpred,j​\(θ⋆,Sj\)−Ztgt,j​\(θ⋆,Sj\),\\displaystyle=Z\_\{\\mathrm\{pred\},j\}\(\\theta^\{\\star\};S\_\{j\}\)\-Z\_\{\\mathrm\{tgt\},j\}\(\\theta^\{\\star\};S\_\{j\}\),\(28\)rjp\\displaystyle r\_\{j\}^\{\\mathrm\{p\}\}=Zpred,j​\(θ⋆,S^j\)−Ztgt,j​\(θ⋆,S^j\),\\displaystyle=Z\_\{\\mathrm\{pred\},j\}\(\\theta^\{\\star\};\\widehat\{S\}\_\{j\}\)\-Z\_\{\\mathrm\{tgt\},j\}\(\\theta^\{\\star\};\\widehat\{S\}\_\{j\}\),ej\\displaystyle e\_\{j\}=rjp−rjc=ejctx−ejtgt,\\displaystyle=r\_\{j\}^\{\\mathrm\{p\}\}\-r\_\{j\}^\{\\mathrm\{c\}\}=e\_\{j\}^\{\\mathrm\{ctx\}\}\-e\_\{j\}^\{\\mathrm\{tgt\}\},ejctx\\displaystyle e\_\{j\}^\{\\mathrm\{ctx\}\}=Zpred,j​\(θ⋆,S^j\)\\displaystyle=Z\_\{\\mathrm\{pred\},j\}\(\\theta^\{\\star\};\\widehat\{S\}\_\{j\}\)−Zpred,j​\(θ⋆,Sj\),\\displaystyle\-Z\_\{\\mathrm\{pred\},j\}\(\\theta^\{\\star\};S\_\{j\}\),ejtgt\\displaystyle e\_\{j\}^\{\\mathrm\{tgt\}\}=Ztgt,j​\(θ⋆,S^j\)\\displaystyle=Z\_\{\\mathrm\{tgt\},j\}\(\\theta^\{\\star\};\\widehat\{S\}\_\{j\}\)−Ztgt,j​\(θ⋆,Sj\)\.\\displaystyle\-Z\_\{\\mathrm\{tgt\},j\}\(\\theta^\{\\star\};S\_\{j\}\)\.This decomposition is not a binary assignment of an error to the context or target route\. A changed pseudo\-token can affect the visible predictor context in one mask realization and the clean target states of other positions in the same example\. The indexjjmay therefore include sampling over mask realizations and masked positions\.

LetAj=∂rj​\(θ,Sj\)/∂θ\|θ⋆A\_\{j\}=\\partial r\_\{j\}\(\\theta;S\_\{j\}\)/\\partial\\theta\|\_\{\\theta^\{\\star\}\}andA^j=∂rj​\(θ,S^j\)/∂θ\|θ⋆\\widehat\{A\}\_\{j\}=\\partial r\_\{j\}\(\\theta;\\widehat\{S\}\_\{j\}\)/\\partial\\theta\|\_\{\\theta^\{\\star\}\}\. These derivatives follow the implemented MSE path, so the detached target contributes to the value ofeje\_\{j\}but not to either residual Jacobian\. The exact gradient difference for the squared residual is

∇θℓj​\(θ⋆,S^j\)−∇θℓj​\(θ⋆,Sj\)\\displaystyle\\nabla\_\{\\theta\}\\ell\_\{j\}\(\\theta^\{\\star\};\\widehat\{S\}\_\{j\}\)\-\\nabla\_\{\\theta\}\\ell\_\{j\}\(\\theta^\{\\star\};S\_\{j\}\)=A^j⊤​rjp−Aj⊤​rjc\\displaystyle=\\widehat\{A\}\_\{j\}^\{\\top\}r\_\{j\}^\{\\mathrm\{p\}\}\-A\_\{j\}^\{\\top\}r\_\{j\}^\{\\mathrm\{c\}\}\(29\)=Aj⊤​ej\+ρj,\\displaystyle=A\_\{j\}^\{\\top\}e\_\{j\}\+\\rho\_\{j\},ρj\\displaystyle\\rho\_\{j\}=\(A^j−Aj\)⊤​rjp\.\\displaystyle=\(\\widehat\{A\}\_\{j\}\-A\_\{j\}\)^\{\\top\}r\_\{j\}^\{\\mathrm\{p\}\}\.The leading term pulls the explicit latent residual perturbation back to parameter space\. The remainderρj\\rho\_\{j\}records the change of the predictor Jacobian caused by the pseudo sequence and becomes small when the local shared\-Jacobian approximation is accurate or the perturbed residual is small\.

LetK=∇θ2​ℒclean​\(θ⋆\)K=\\nabla\_\{\\theta\}^\{2\}\\mathcal\{L\}\_\{\\mathrm\{clean\}\}\(\\theta^\{\\star\}\)be positive definite on the local parameter subspace under analysis, and setu=θ−θ⋆u=\\theta\-\\theta^\{\\star\}\. Neglectingρj\\rho\_\{j\}and the change in curvature between the two objectives gives

ℒpseudo​\(θ⋆\+u\)−ℒpseudo​\(θ⋆\)≈12​u⊤​K​u\+λn​∑j=1nej⊤​Aj​u\.\\mathcal\{L\}\_\{\\mathrm\{pseudo\}\}\(\\theta^\{\\star\}\+u\)\-\\mathcal\{L\}\_\{\\mathrm\{pseudo\}\}\(\\theta^\{\\star\}\)\\approx\\frac\{1\}\{2\}u^\{\\top\}Ku\+\\frac\{\\lambda\}\{n\}\\sum\_\{j=1\}^\{n\}e\_\{j\}^\{\\top\}A\_\{j\}u\.\(30\)Its stationary displacement is

upseudo=−λnK−1∑j=1nAj⊤ej\.u\_\{\\mathrm\{pseudo\}\}=\-\\frac\{\\lambda\}\{n\}K^\{\-1\}\\sum\_\{j=1\}^\{n\}A\_\{j\}^\{\\top\}e\_\{j\}\.\(31\)This is a change in the shared model parameters, not an independently optimized hidden state\. Its effect on any clean example is mediated by that example’s parameter\-to\-state Jacobian\.

###### Proof of[Theorem3\.2](https://arxiv.org/html/2610.36493#S3.Thmproposition2)\.

For a clean evaluation contextxx, defineJx=∂Hx/∂θ\|θ⋆J\_\{x\}=\\partial H\_\{x\}/\\partial\\theta\|\_\{\\theta^\{\\star\}\}anddx,y​k=∇HxGx,y​k​\(θ⋆\)d\_\{x,yk\}=\\nabla\_\{H\_\{x\}\}G\_\{x,yk\}\(\\theta^\{\\star\}\)\. The corresponding parameter\-space margin gradient isqx,y​k=Jx⊤​dx,y​kq\_\{x,yk\}=J\_\{x\}^\{\\top\}d\_\{x,yk\}\. Combining the first\-order margin expansion with[Equation31](https://arxiv.org/html/2610.36493#A1.E31)gives

Gx,y​klin\\displaystyle G\_\{x,yk\}^\{\\mathrm\{lin\}\}=mx,y​k\+qx,y​k⊤​upseudo\\displaystyle=m\_\{x,yk\}\+q\_\{x,yk\}^\{\\top\}u\_\{\\mathrm\{pseudo\}\}\(32\)=mx,y​k−λn​∑j=1ndx,y​k⊤​Jx​K−1​Aj⊤​ej\\displaystyle=m\_\{x,yk\}\-\\frac\{\\lambda\}\{n\}\\sum\_\{j=1\}^\{n\}d\_\{x,yk\}^\{\\top\}J\_\{x\}K^\{\-1\}A\_\{j\}^\{\\top\}e\_\{j\}=mx,y​k−λζ¯x,y​k,ζ¯x,y​k=1n∑j=1nζx,y​k,j\.\\displaystyle=m\_\{x,yk\}\-\\lambda\\overline\{\\zeta\}\_\{x,yk\},\\qquad\\overline\{\\zeta\}\_\{x,yk\}=\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}\\zeta\_\{x,yk,j\}\.Under the theorem assumptions,𝔼⁡\[ζ¯x,y​k\]=βx,y​k\\mathbb\{E\}\[\\overline\{\\zeta\}\_\{x,yk\}\]=\\beta\_\{x,yk\}andVar⁡\(ζ¯x,y​k\)≤σx,y​k2/n\\operatorname\{Var\}\(\\overline\{\\zeta\}\_\{x,yk\}\)\\leq\\sigma\_\{x,yk\}^\{2\}/n\. Whenmx,y​k\>λ​βx,y​km\_\{x,yk\}\>\\lambda\\beta\_\{x,yk\}, boundary crossing requires

ζ¯x,y​k−βx,y​k≥mx,y​k−λ​βx,y​kλ\.\\overline\{\\zeta\}\_\{x,yk\}\-\\beta\_\{x,yk\}\\geq\\frac\{m\_\{x,yk\}\-\\lambda\\beta\_\{x,yk\}\}\{\\lambda\}\.\(33\)Cantelli’s inequality therefore yields

Pr\[Gx,y​klin≤0\]≤σx,y​k2/nσx,y​k2/n\+\(mx,y​k/λ−βx,y​k\)2,\\Pr\[G\_\{x,yk\}^\{\\mathrm\{lin\}\}\\leq 0\]\\leq\\frac\{\\sigma\_\{x,yk\}^\{2\}/n\}\{\\sigma\_\{x,yk\}^\{2\}/n\+\(m\_\{x,yk\}/\\lambda\-\\beta\_\{x,yk\}\)^\{2\}\},\(34\)which is equivalent to[Equation12](https://arxiv.org/html/2610.36493#S3.E12)\. ∎

For a multi\-class next\-token decision, applying the union bound overk∈𝒱dec∖\{y\}k\\in\\mathcal\{V\}\_\{\\mathrm\{dec\}\}\\setminus\\\{y\\\}gives

Pr\[argmaxsax,s\(θ\)≠y\]≤∑k≠yPr\[Gx,y​k\(θ\)≤0\]\.\\Pr\[\\operatorname\*\{argmax\}\_\{s\}a\_\{x,s\}\(\\theta\)\\neq y\]\\leq\\sum\_\{k\\neq y\}\\Pr\[G\_\{x,yk\}\(\\theta\)\\leq 0\]\.\(35\)Each term uses its own clean margin, output\-influence bias, and variance; no independence across competitors is required\.

Approximation terms and SIGReg\.

Equation \([29](https://arxiv.org/html/2610.36493#A1.E29)\) makes the shared\-Jacobian approximation explicit\. Retaining its remainder changes[Equation31](https://arxiv.org/html/2610.36493#A1.E31)by−λK−1n−1∑jρj\-\\lambda K^\{\-1\}n^\{\-1\}\\sum\_\{j\}\\rho\_\{j\}\. The pseudo sequence also changes the clean latents entering modality\-wise SIGReg\. Ifgsigg\_\{\\mathrm\{sig\}\}denotes the difference between the pseudo\-sequence and true\-sequence SIGReg gradients, the total additional displacement is

urem=−K−1​\(λn​∑j=1nρj\+gsig\)\.u\_\{\\mathrm\{rem\}\}=\-K^\{\-1\}\\left\(\\frac\{\\lambda\}\{n\}\\sum\_\{j=1\}^\{n\}\\rho\_\{j\}\+g\_\{\\mathrm\{sig\}\}\\right\)\.\(36\)We keep these terms separate fromeje\_\{j\}because they do not describe a latent residual perturbation\. They can be incorporated into[Theorem3\.2](https://arxiv.org/html/2610.36493#S3.Thmproposition2)by addingqx,y​k⊤​uremq\_\{x,yk\}^\{\\top\}u\_\{\\mathrm\{rem\}\}to the margin and controlling its bias and variance\. SIGReg can contribute stabilizing curvature toKK, but its pseudo\-label\-dependent gradient is not assumed to be zero\.

The nonlinear parameter\-to\-margin map contributes a Taylor remainder\. If the combined scalar remainder inGx,y​k=mx,y​k−λ​ζ¯x,y​k\+ℛx,y​kG\_\{x,yk\}=m\_\{x,yk\}\-\\lambda\\overline\{\\zeta\}\_\{x,yk\}\+\\mathcal\{R\}\_\{x,yk\}satisfies\|ℛx,y​k\|≤τx,y​k\|\\mathcal\{R\}\_\{x,yk\}\|\\leq\\tau\_\{x,yk\}, the same argument gives

Pr\[Gx,y​k≤0\]≤λ2​σx,y​k2/nλ2​σx,y​k2/n\+\(mx,y​k−τx,y​k−λ​βx,y​k\)2\\Pr\[G\_\{x,yk\}\\leq 0\]\\leq\\frac\{\\lambda^\{2\}\\sigma\_\{x,yk\}^\{2\}/n\}\{\\lambda^\{2\}\\sigma\_\{x,yk\}^\{2\}/n\+\(m\_\{x,yk\}\-\\tau\_\{x,yk\}\-\\lambda\\beta\_\{x,yk\}\)^\{2\}\}\(37\)whenevermx,y​k\>τx,y​k\+λ​βx,y​km\_\{x,yk\}\>\\tau\_\{x,yk\}\+\\lambda\\beta\_\{x,yk\}\.

##### Anchor curvature\.

The parameter\-space curvature makes the roles of SFT and JEPA explicit\. LetWi=∂ai/∂θW\_\{i\}=\\partial a\_\{i\}/\\partial\\thetabe the Jacobian of the clean SFT logitsaia\_\{i\}and letpip\_\{i\}be their token distribution\. The cross\-entropy Gauss–Newton term is

KSFT,i=Wi⊤​\[Diag⁡\(pi\)−pi​pi⊤\]​Wi⪰0\.K\_\{\\mathrm\{SFT\},i\}=W\_\{i\}^\{\\top\}\[\\operatorname\{Diag\}\(p\_\{i\}\)\-p\_\{i\}p\_\{i\}^\{\\top\}\]W\_\{i\}\\succeq 0\.\(38\)IfBjB\_\{j\}is the parameter Jacobian of a clean JEPA residual, its squared loss contributesBj⊤​BjB\_\{j\}^\{\\top\}B\_\{j\}\. A damped local Gauss–Newton model on an identifiable parameter subspace𝒰\\mathcal\{U\}is

K\\displaystyle K≈αS​∑iKSFT,i\+αJ​∑jBj⊤​Bj\\displaystyle\\approx\\alpha\_\{\\mathrm\{S\}\}\\sum\_\{i\}K\_\{\\mathrm\{SFT\},i\}\+\\alpha\_\{\\mathrm\{J\}\}\\sum\_\{j\}B\_\{j\}^\{\\top\}B\_\{j\}\(39\)\+Kreg\+μ​P𝒰,\\displaystyle\+K\_\{\\mathrm\{reg\}\}\+\\mu P\_\{\\mathcal\{U\}\},Kreg\\displaystyle K\_\{\\mathrm\{reg\}\}⪰0,μ\>0\.\\displaystyle\\succeq 0,\\qquad\\mu\>0\.whereKregK\_\{\\mathrm\{reg\}\}is a positive\-semidefinite local approximation to regularization curvature andP𝒰P\_\{\\mathcal\{U\}\}projects onto𝒰\\mathcal\{U\}\. The restriction is necessary because neural\-network parameterizations contain flat and symmetry\-related directions\. The theorem depends on curvature only along the pulled\-back pseudo\-error and evaluation\-margin directions, rather than on global positive definiteness of the full network Hessian\. With mean\-reduced objectives, more clean samples do not automatically scaleKK; they can nevertheless improve curvature coverage and reduce its estimation variance\.

##### Bias, variance, and dependence\.

The harmful bias is

βx,y​k=𝔼⁡\[dx,y​k⊤​Jx​K−1​Aj⊤​ej\],\\beta\_\{x,yk\}=\\mathbb\{E\}\[d\_\{x,yk\}^\{\\top\}J\_\{x\}K^\{\-1\}A\_\{j\}^\{\\top\}e\_\{j\}\],\(40\)which depends jointly on the latent residual perturbation, the training Jacobian, the anchored curvature, and the evaluation readout\. Consequently,𝔼⁡\[ej\]=0\\mathbb\{E\}\[e\_\{j\}\]=0is not sufficient whenAjA\_\{j\}andeje\_\{j\}are dependent\. The stronger condition𝔼⁡\[ej∣Aj\]=0\\mathbb\{E\}\[e\_\{j\}\\mid A\_\{j\}\]=0impliesβx,y​k=0\\beta\_\{x,yk\}=0for fixedxx,KK, andθ⋆\\theta^\{\\star\}, but model\-generated pseudo\-labels need not satisfy it\. The bias–variance statement in[Theorem3\.2](https://arxiv.org/html/2610.36493#S3.Thmproposition2)therefore retainsβx,y​k\\beta\_\{x,yk\}instead of assuming exact centering\. If pseudo constraints are correlated, the variance of their average becomesn−2​∑j,j′Cov⁡\(ζj,ζj′\)n^\{\-2\}\\sum\_\{j,j^\{\\prime\}\}\\operatorname\{Cov\}\(\\zeta\_\{j\},\\zeta\_\{j^\{\\prime\}\}\)and need not decay as1/n1/n\.

##### Single\-error comparison with direct pseudo\-SFT\.

For one erroneous pseudo\-token, lety~\\widetilde\{y\}be the erroneous alternative,Gy​y~​\(θ⋆\)=m\>0G\_\{y\\widetilde\{y\}\}\(\\theta^\{\\star\}\)=m\>0, andq=∇θGy​y~​\(θ⋆\)q=\\nabla\_\{\\theta\}G\_\{y\\widetilde\{y\}\}\(\\theta^\{\\star\}\)\. One gradient step of sizeρ\\rhoon the pseudo\-JEPA perturbation relative to its clean\-pair counterpart gives

θJ′=θ⋆−ρ​λ​A⊤​e,mJ′≈m−ρ​λ​q⊤​A⊤​e\.\\theta\_\{\\mathrm\{J\}\}^\{\\prime\}=\\theta^\{\\star\}\-\\rho\\lambda A^\{\\top\}e,\\qquad m\_\{\\mathrm\{J\}\}^\{\\prime\}\\approx m\-\\rho\\lambda q^\{\\top\}A^\{\\top\}e\.\(41\)In a binary local readout, letpy=sigmoid⁡\(m\)p\_\{y\}=\\operatorname\{sigmoid\}\(m\)\. Cross\-entropy towardy~\\widetilde\{y\}has gradientpy​qp\_\{y\}q, so direct pseudo\-SFT gives

θS′=θ⋆−ρ​λ​py​q,mS′≈m−ρ​λ​py​‖q‖22\.\\theta\_\{\\mathrm\{S\}\}^\{\\prime\}=\\theta^\{\\star\}\-\\rho\\lambda p\_\{y\}q,\\qquad m\_\{\\mathrm\{S\}\}^\{\\prime\}\\approx m\-\\rho\\lambda p\_\{y\}\\\|q\\\|\_\{2\}^\{2\}\.\(42\)Thus the wrong categorical target decreases the correct\-versus\-wrong margin in this local model\. The pseudo\-JEPA effect instead depends on the signed alignmentq⊤​A⊤​eq^\{\\top\}A^\{\\top\}eand can either decrease or increase that margin\. For the corresponding binary probability,

\|pyJ−py\|≤ρ​λ4​\|q⊤​A⊤​e\|≤ρ​λ4​‖A​q‖2​‖e‖2\.\|p\_\{y\}^\{\\mathrm\{J\}\}\-p\_\{y\}\|\\leq\\frac\{\\rho\\lambda\}\{4\}\|q^\{\\top\}A^\{\\top\}e\|\\leq\\frac\{\\rho\\lambda\}\{4\}\\\|Aq\\\|\_\{2\}\\\|e\\\|\_\{2\}\.\(43\)
With the clean anchor retained, the local pseudo\-JEPA equilibrium isuJ=−λ​K−1​A⊤​eu\_\{\\mathrm\{J\}\}=\-\\lambda K^\{\-1\}A^\{\\top\}e\. A quadratic expansion of the erroneous pseudo\-SFT term gives

uS≈−λ​py​\[K\+λ​py​\(1−py\)​q​q⊤\]−1​q\.u\_\{\\mathrm\{S\}\}\\approx\-\\lambda p\_\{y\}\[K\+\\lambda p\_\{y\}\(1\-p\_\{y\}\)qq^\{\\top\}\]^\{\-1\}q\.\(44\)The bracketed matrix is positive definite on𝒰\\mathcal\{U\}, andq⊤​uS<0q^\{\\top\}u\_\{\\mathrm\{S\}\}<0\. This comparison does not order the magnitudes of all JEPA and SFT errors\. It identifies their different routes to the output boundary: pseudo\-SFT directly optimizes an erroneous categorical preference, whereas pseudo\-JEPA first induces a continuous residual perturbation whose effect is filtered byAA,K−1K^\{\-1\}, and the clean readout direction\.

#### A\.5\.3Multimodal Evidence and Local Readout Stability

[Theorem3\.1](https://arxiv.org/html/2610.36493#S3.Thmproposition1)relates decoder\-relevant latent variation to next\-token stability, but it does not quantify how evidence from multiple modalities changes that variation\. We considerMMco\-observed modalities that describe the same underlying entity or state\. For modalitymm, let𝒢m\\mathcal\{G\}\_\{m\}contain its reliable, target\-relevant entries, and let𝒩\\mathcal\{N\}contain the visible noisy entries across the multimodal context\. Under a locally fixed attention map with scoreαk\\alpha\_\{k\}and temperatureτ\>0\\tau\>0, define

Zm\\displaystyle Z\_\{m\}=∑g∈𝒢meαg/τ,\\displaystyle=\\sum\_\{g\\in\\mathcal\{G\}\_\{m\}\}e^\{\\alpha\_\{g\}/\\tau\},Z𝒩\\displaystyle Z\_\{\\mathcal\{N\}\}=∑j∈𝒩eαj/τ,\\displaystyle=\\sum\_\{j\\in\\mathcal\{N\}\}e^\{\\alpha\_\{j\}/\\tau\},\(45\)Z𝒢\(M\)\\displaystyle Z\_\{\\mathcal\{G\}\}^\{\(M\)\}=∑m=1MZm,\\displaystyle=\\sum\_\{m=1\}^\{M\}Z\_\{m\},ΔM\\displaystyle\\Delta\_\{M\}=log⁡Z𝒢\(M\)Z𝒩\.\\displaystyle=\\log\\frac\{Z\_\{\\mathcal\{G\}\}^\{\(M\)\}\}\{Z\_\{\\mathcal\{N\}\}\}\.Letak∝eαk/τa\_\{k\}\\propto e^\{\\alpha\_\{k\}/\\tau\}be the corresponding attention weights\. The total noisy attention mass and its effective support are

βM=∑j∈𝒩aj=11\+eΔM,neff=βM2∑j∈𝒩aj2\.\\beta\_\{M\}=\\sum\_\{j\\in\\mathcal\{N\}\}a\_\{j\}=\\frac\{1\}\{1\+e^\{\\Delta\_\{M\}\}\},\\qquad n\_\{\\mathrm\{eff\}\}=\\frac\{\\beta\_\{M\}^\{2\}\}\{\\sum\_\{j\\in\\mathcal\{N\}\}a\_\{j\}^\{2\}\}\.\(46\)
Suppose only the noisy values are perturbed,v~j=vj\+ξj\\widetilde\{v\}\_\{j\}=v\_\{j\}\+\\xi\_\{j\}forj∈𝒩j\\in\\mathcal\{N\}, where theξj\\xi\_\{j\}are conditionally independent, centered, and satisfyCov⁡\(ξj∣Ci,s\+\)⪯σv2​I\\operatorname\{Cov\}\(\\xi\_\{j\}\\mid C\_\{i\},s^\{\+\}\)\\preceq\\sigma\_\{v\}^\{2\}I\. Let their attention\-output perturbation beo~M−oM⋆=∑j∈𝒩aj​ξj\\widetilde\{o\}\_\{M\}\-o\_\{M\}^\{\\star\}=\\sum\_\{j\\in\\mathcal\{N\}\}a\_\{j\}\\xi\_\{j\}\. We locally decompose the correct\-versus\-competitor margin as

Δ~i,s−=μi,s−\+ε0\+w⊤​\(o~M−oM⋆\),\\widetilde\{\\Delta\}\_\{i,s^\{\-\}\}=\\mu\_\{i,s^\{\-\}\}\+\\varepsilon\_\{0\}\+w^\{\\top\}\(\\widetilde\{o\}\_\{M\}\-o\_\{M\}^\{\\star\}\),\(47\)whereμi,s−\>0\\mu\_\{i,s^\{\-\}\}\>0,ε0\\varepsilon\_\{0\}is conditionally centered and uncorrelated with theξj\\xi\_\{j\}, andVar⁡\(ε0∣Ci,s\+\)≤v0\\operatorname\{Var\}\(\\varepsilon\_\{0\}\\mid C\_\{i\},s^\{\+\}\)\\leq v\_\{0\}\.

###### Corollary A\.1\(Multimodal evidence tightens the local readout bound\)\.

Under the conditions above, define

BMvis=‖w‖22​σv2neff​\(1\+eΔM\)2\.B\_\{M\}^\{\\mathrm\{vis\}\}=\\frac\{\\\|w\\\|\_\{2\}^\{2\}\\sigma\_\{v\}^\{2\}\}\{n\_\{\\mathrm\{eff\}\}\(1\+e^\{\\Delta\_\{M\}\}\)^\{2\}\}\.\(48\)Then

Pr\[Δ~i,s−≤0∣Ci,s\+\]≤v0\+BMvisv0\+BMvis\+μi,s−2\.\\Pr\[\\widetilde\{\\Delta\}\_\{i,s^\{\-\}\}\\leq 0\\mid C\_\{i\},s^\{\+\}\]\\leq\\frac\{v\_\{0\}\+B\_\{M\}^\{\\mathrm\{vis\}\}\}\{v\_\{0\}\+B\_\{M\}^\{\\mathrm\{vis\}\}\+\\mu\_\{i,s^\{\-\}\}^\{2\}\}\.\(49\)If an additional modality contributes reliable evidence massZM\+1\>0Z\_\{M\+1\}\>0while the existing scores, noisy entries, and local readout map remain fixed, then

BM\+1visBMvis=\[1\+eΔM1\+eΔM​\(1\+ZM\+1/Z𝒢\(M\)\)\]2<1\.\\frac\{B\_\{M\+1\}^\{\\mathrm\{vis\}\}\}\{B\_\{M\}^\{\\mathrm\{vis\}\}\}=\\left\[\\frac\{1\+e^\{\\Delta\_\{M\}\}\}\{1\+e^\{\\Delta\_\{M\}\}\(1\+Z\_\{M\+1\}/Z\_\{\\mathcal\{G\}\}^\{\(M\)\}\)\}\\right\]^\{2\}<1\.\(50\)If every modality suppliesZm≥Zmin\>0Z\_\{m\}\\geq Z\_\{\\min\}\>0, then

BMvis≤‖w‖22​σv2neff​\(1\+M​ZminZ𝒩\)−2\.B\_\{M\}^\{\\mathrm\{vis\}\}\\leq\\frac\{\\\|w\\\|\_\{2\}^\{2\}\\sigma\_\{v\}^\{2\}\}\{n\_\{\\mathrm\{eff\}\}\}\\left\(1\+\\frac\{MZ\_\{\\min\}\}\{Z\_\{\\mathcal\{N\}\}\}\\right\)^\{\-2\}\.\(51\)Thus the local decision\-reversal bound tightens monotonically withMMwhen added modalities preserve the mean margin, do not increasev0v\_\{0\}, and contribute reliable evidence under the fixed local map\.

The corollary characterizes multimodal redundancy as evidence quality rather than modality count alone\. Images, audio, language, numerical sensor streams, and radar measurements can provide complementary observations of the same underlying state\. When each added modality contributes aligned evidence, their aggregate partition mass increasesΔM\\Delta\_\{M\}and reduces the variance transmitted from visible noisy entries\. This is the precise sense in which additional modalities can strengthen mutual verification and improve local noise tolerance\.

The monotonic result is conditional\. A redundant, mismatched, systematically biased, or ignored modality need not preserve the mean margin or contribute reliable evidence mass\. The analysis also concerns a local next\-token decision under fixed attention scores and centered independent perturbations\. It therefore does not guarantee that adding modalities monotonically improves full\-sequence exact\-match accuracy\. The image–symbol setting studied by AC\-JEPA is theM=2M=2instance of this more general mechanism\.

###### Proof\.

Conditional centering makes the attention\-output perturbation mean\-zero\. WriteΣj=Cov⁡\(ξj∣Ci,s\+\)\\Sigma\_\{j\}=\\operatorname\{Cov\}\(\\xi\_\{j\}\\mid C\_\{i\},s^\{\+\}\)\. For anyww,

Var\[w⊤\(o~M−oM⋆\)∣Ci,s\+\]\\displaystyle\\mathrm\{Var\}\\\!\\left\[w^\{\\top\}\(\\widetilde\{o\}\_\{M\}\-o\_\{M\}^\{\\star\}\)\\mid C\_\{i\},s^\{\+\}\\right\]=∑j∈𝒩aj2​w⊤​Σj​w\\displaystyle=\\sum\_\{j\\in\\mathcal\{N\}\}a\_\{j\}^\{2\}w^\{\\top\}\\Sigma\_\{j\}w\(52\)≤‖w‖22​σv2​∑j∈𝒩aj2\\displaystyle\\leq\\\|w\\\|\_\{2\}^\{2\}\\sigma\_\{v\}^\{2\}\\sum\_\{j\\in\\mathcal\{N\}\}a\_\{j\}^\{2\}=‖w‖22​σv2​βM2neff=BMvis\.\\displaystyle=\\frac\{\\\|w\\\|\_\{2\}^\{2\}\\sigma\_\{v\}^\{2\}\\beta\_\{M\}^\{2\}\}\{n\_\{\\mathrm\{eff\}\}\}=B\_\{M\}^\{\\mathrm\{vis\}\}\.Adding the uncorrelated residual variancev0v\_\{0\}and applying the first inequality in[Theorem3\.1](https://arxiv.org/html/2610.36493#S3.Thmproposition1)proves[Equation49](https://arxiv.org/html/2610.36493#A1.E49)\.

Adding modalityM\+1M\+1replacesZ𝒢\(M\)Z\_\{\\mathcal\{G\}\}^\{\(M\)\}byZ𝒢\(M\)\+ZM\+1Z\_\{\\mathcal\{G\}\}^\{\(M\)\}\+Z\_\{M\+1\}and therefore gives

ΔM\+1=ΔM\+log⁡\(1\+ZM\+1Z𝒢\(M\)\)\.\\Delta\_\{M\+1\}=\\Delta\_\{M\}\+\\log\\left\(1\+\\frac\{Z\_\{M\+1\}\}\{Z\_\{\\mathcal\{G\}\}^\{\(M\)\}\}\\right\)\.\(53\)All noisy attention weights are scaled by the same factor, soneffn\_\{\\mathrm\{eff\}\}remains unchanged\. Substitution into[Equation48](https://arxiv.org/html/2610.36493#A1.E48)yields[Equation50](https://arxiv.org/html/2610.36493#A1.E50)\. Finally,Z𝒢\(M\)=∑mZm≥M​ZminZ\_\{\\mathcal\{G\}\}^\{\(M\)\}=\\sum\_\{m\}Z\_\{m\}\\geq MZ\_\{\\min\}, which proves[Equation51](https://arxiv.org/html/2610.36493#A1.E51)\. ∎